From acfdf42a9a9e91e35859dd3a32184941d0207d8c Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 15:41:06 +0200 Subject: [PATCH 01/10] fix(DEV-1756): bound generated identifiers to the dialect's length limit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Postgres caps identifiers at 63 bytes and truncates past it SILENTLY (a NOTICE, never an error). SLayer's alias convention `..` crosses that on a 3-hop join, so two sibling aliases collapse onto one output name: SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name 73 B SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email 74 B Under the DEV-1444 outer wrap that raises AmbiguousColumnError; with no sibling to collide with the query succeeds and the column silently disappears, which is the worse case. Every dialect now declares a conservative `max_identifier_bytes` budget, and an over-limit identifier is fitted at emission to `__` (new `slayer/sql/dialects/_identifier_fit.py`). Fitting is a pure function of the full original name, so the read side rebuilds the emitted->canonical map by re-running it — nothing is threaded through generation — and `decode_result_keys` restores the canonical dotted alias. Consumers never see the shortened form; only `response.sql` does, because that is the SQL that ran. Shortened only when over the limit, so under-limit output stays byte-identical (pinned by pre-change goldens, not merely by idempotence). Both ends of the alias survive because the reported colliding pair differs only in its final segment. The write pass is an exact-match replacement over the query's own alias set (`all_projection_aliases`, unfiltered — hidden ORDER-BY hoists are projected in the inner SELECT and truncate identically), never a length regex over arbitrary SQL, so a long quoted-looking span inside a string literal cannot be corrupted. Substitution is two-phase as defence-in-depth. BigQuery/T-SQL run their existing dot-mangle regex after the length pass: an under-limit alias makes that pass a no-op so their output is unchanged, and an over-limit one arrives still-dotted and is mangled by the same regex — no double-encoding — with the budget sized against `encode_alias`. Covers the three output-name surfaces: projection aliases, CTE names (allocated through a per-statement collision-checked `SQLGenerator._cte_name`), and `_query_as_model` virtual-model shorts. Join-path TABLE aliases are deferred to DEV-1743, which already owns the `__` path-alias allocator. Collisions raise `IdentifierCollisionError` rather than emitting ambiguous SQL; the check covers identity entries too, since an already-short alias equal to another's fitted form is a duplicate no hash width can prevent. Also fixes a pre-existing bug in the same code path, reproduced on a live server: `_query_as_model` emitted its short alias bare, so Postgres case-folded it while the outer stage referenced it quoted, making any query-backed model with a mixed-case join path unqueryable (UndefinedColumnError). It is now always dialect-quoted, which additionally takes these names out of the case-folding namespace. Co-Authored-By: Claude Opus 5 (1M context) --- DECISIONS.md | 1 + docs/concepts/queries.md | 8 + docs/database-support.md | 61 ++ slayer/core/errors.py | 38 + slayer/engine/enriched.py | 23 + slayer/engine/query_engine.py | 75 +- slayer/sql/dialects/_identifier_fit.py | 163 ++++ slayer/sql/dialects/_tier2.py | 12 + slayer/sql/dialects/base.py | 154 +++- slayer/sql/dialects/bigquery.py | 54 +- slayer/sql/dialects/clickhouse.py | 2 + slayer/sql/dialects/duckdb.py | 2 + slayer/sql/dialects/mysql.py | 2 + slayer/sql/dialects/postgres.py | 2 + slayer/sql/dialects/snowflake.py | 2 + slayer/sql/dialects/sqlite.py | 2 + slayer/sql/dialects/tsql.py | 48 +- slayer/sql/generator.py | 81 +- tests/dialects/test_bigquery.py | 4 +- tests/dialects/test_identifier_fit.py | 502 ++++++++++ .../test_dev1756_identifier_length_pg.py | 307 +++++++ tests/test_dev1756_identifier_length.py | 868 ++++++++++++++++++ tests/test_query_backed_models.py | 15 +- tests/test_sql_generator.py | 4 +- 24 files changed, 2361 insertions(+), 69 deletions(-) create mode 100644 slayer/sql/dialects/_identifier_fit.py create mode 100644 tests/dialects/test_identifier_fit.py create mode 100644 tests/integration/test_dev1756_identifier_length_pg.py create mode 100644 tests/test_dev1756_identifier_length.py diff --git a/DECISIONS.md b/DECISIONS.md index 496c8b6f..1d8cb78b 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -69,3 +69,4 @@ implementation detail. Include issue refs when known. - 2026-08-03 — Optional blocks + Cube JS/FILTER_PARAMS import (DEV-1730 / #270): a Mode-A-only `{? ... ?}` block renders its content parenthesised when every inner `{var}` is supplied, else collapses to the neutral `(1=1)` — the SLayer form of a Cube `FILTER_PARAMS` optional pushdown. Blocks live in the same `substitute_variables` (escape="sql") scanner as `{var}`/`{{`/`}}`, must contain ≥1 var, do not nest, and are rejected in Mode-B. A block-bearing model runs substitution even on a zero-variable call so its blocks collapse (the `_substitute_model_sql_surfaces` fast-path now checks for `{?` too); a block-free, required-only model with zero variables is still left untouched (the documented DEV-1625 raw-brace-literal boundary). `extract_model_variables(model)` derives required (bare, no default) vs optional (in-block or defaulted) from the four Mode-A surfaces — structural, nothing persisted, surfaced additively in the inspect skeleton `Variables:` line. The Cube importer gains a **JavaScript front-end** (esprima ESTree parser, a new core dep) that parses `cube()`/`view()` into the same `CubeCube`/`CubeView` shapes as YAML (dynamic values → report + skip member). FILTER_PARAMS refs are carried JS→converter as structured `CubeFilterParamRef` on the transient `CubeCube` (sentinels in the surface text; no arrow-body re-parse, sidestepping the `{var}`-vs-`{FILTER_PARAMS…}` brace clash); the converter resolves sentinels AFTER `translate_cube_refs` so the introduced `{var}` are never eaten. Requiredness (bare vs block) is decided in the converter alone via `honor_required_meta` (default on; CLI `--ignore-required-meta`) AND the member's `meta.required`; with the flag off a scalar-position arrow collapses to Cube's own `(1=1)::TIMESTAMP` booby-trap, faithfully. Cross-cube refs, unknown members, and generated-name collisions (`d`→`d_from` clashing member `d_from`) drop the cube (`filter_params_unsupported`); each logical variable is reported once (`filter_params_variable`) and stashed in `meta.cube_variables`. `render_probe_text` (blocks→`(1=1)`, bare vars→`0`) is the single import-time validation renderer, matching runtime collapse. - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. +- 2026-08-06 — Dialect-aware identifier-length fitting (DEV-1756): every dialect declares a conservative universal budget as `SqlDialect.max_identifier_bytes` (postgres 63, mysql 64, redshift 127, oracle/tsql 128, snowflake 255, duckdb 256, bigquery 300, `None` = unbounded for sqlite/clickhouse/trino/presto/databricks/spark), and an over-limit identifier is shortened at emission to `__` via `slayer/sql/dialects/_identifier_fit.py`. Postgres is the binding case and the reason this is a correctness fix rather than cosmetics: it truncates over-length identifiers **silently** (a NOTICE, never an error), so two 3-hop aliases sharing a 63-byte prefix either blow up as `AmbiguousColumnError` under the DEV-1444 outer wrap or — with no sibling to collide with — quietly return the column under a name the engine never looks up. Shorten **only when over the limit**, never uniformly: `decode_result_keys` restores the canonical dotted alias, so the dialect-dependence the issue worried about is invisible to consumers and the 99% case keeps byte-identical SQL (pinned by pre-change goldens, not merely by idempotence). Both ends of the alias are kept because the reported colliding pair differs ONLY in its final segment — a head-only truncation would render the two indistinguishable in `dry_run` output. The digest is sha256 of the FULL original, which is what makes `fit_identifier` a pure function of the name and lets the read side rebuild the emitted→canonical map by re-running it, with no map threaded through generation (the alternative the issue sketched). Write side is an EXACT-match replacement over the query's own alias set (`all_projection_aliases`, unfiltered — hidden ORDER-BY hoists and `_inner_*`/`_ft*`/`_ts*` entries are projected in the inner SELECT and truncate identically), never a length regex over arbitrary SQL, so a long quoted-looking span inside a string literal can never be corrupted; substitution is two-phase (canonical→sentinel→final) as defence-in-depth, though today's key set (over-limit) and value set (within-limit) are provably disjoint. BigQuery/T-SQL compose by running their existing dot-mangle regex AFTER the base length pass: an under-limit alias makes the length pass a genuine no-op so their output is unchanged, and an over-limit one arrives still-dotted and gets mangled by the same regex — no double-encoding — with the budget sized against `encode_alias` so the post-mangle form still fits. Scope is the three OUTPUT-name surfaces: projection aliases, CTE names (`_cte_name_from_alias` fits the whole result, prefix included, allocated through a per-statement collision-checked `SQLGenerator._cte_name`), and `_query_as_model` virtual-model shorts. Join-path TABLE aliases are deferred to DEV-1743, whose plan already owns the `__` path-alias allocator; their failure mode (silent wrong joins) is worse but the fix requires decoupling `EnrichedDimension.model_name` from the emitted qualifier across ~8 sites that split it back on `__`. Collisions raise `IdentifierCollisionError` rather than emitting ambiguous SQL — the check covers identity entries too, since an already-short alias equal to another's fitted form is a duplicate no hash width can prevent. Fixed alongside, in the same code path: `_query_as_model` now ALWAYS dialect-quotes its short alias instead of only for reserved words — emitted bare, a mixed-case short was case-folded by Postgres while the outer stage referenced it quoted, making any query-backed model with a mixed-case join path unqueryable (`UndefinedColumnError`, reproduced on a live server); quoting also takes these names out of the case-folding namespace, so their uniqueness check is the only one that needs to be case-insensitive. diff --git a/docs/concepts/queries.md b/docs/concepts/queries.md index 055ea85c..1cd6a4b6 100644 --- a/docs/concepts/queries.md +++ b/docs/concepts/queries.md @@ -131,6 +131,14 @@ Query results are returned as a `SlayerResponse`: } ``` +Result keys are always this canonical dotted form, on every backend. Where a +database's own rules force a different spelling in the emitted SQL — BigQuery +and SQL Server reject dotted column aliases, and every engine caps identifier +length — the alias is rewritten on the way out and restored on the way back, so +`data` and `columns` do not vary by dialect. Only `sql` shows the rewritten +form, since that is what actually ran. See +[Database support](../database-support.md#identifier-length-limits). + --- ## Filters diff --git a/docs/database-support.md b/docs/database-support.md index fa652472..671011d5 100644 --- a/docs/database-support.md +++ b/docs/database-support.md @@ -35,6 +35,67 @@ Unit tests for SQL generation; no live-instance verification. Redshift, Trino/Presto (Athena uses the Presto dialect), Databricks/Spark, Oracle. +## Identifier length limits + +SLayer's result-column aliases are model-qualified join paths +(`orders.customers.regions.name`), so on a deep join chain they can grow past +what the database allows for an identifier. Postgres is the tightest of the +Tier-1 set at **63 bytes**, and — unlike every other engine here — it +**truncates silently**, emitting only a `NOTICE`. Two aliases sharing a 63-byte +prefix therefore become the same output column: the query either fails with +`AmbiguousColumnError` or, with no sibling to collide with, quietly returns the +column under a name nothing looks up. + +Each dialect declares a budget as `SqlDialect.max_identifier_bytes`. Anything +longer is shortened at emission to `__` — a deterministic +form that keeps both the root model and the column name readable, with an +8-hex-character SHA-256 of the full original in between: + +``` +orders.customers.regions.districts.neighbourhood_name (over 63 bytes) + -> orders.customers.region_4f2a91c7_ricts.neighbourhood_name +``` + +Two properties matter for consumers: + +- **Result keys never change.** The shortening is reversed on the way back, so + `response.data` and `response.columns` always carry the canonical dotted + alias regardless of which backend ran the query. Only `response.sql` (and + `dry_run` / `EXPLAIN` output) shows the shortened form, because that is the + SQL that actually executed. +- **Nothing is shortened unless it must be.** An alias that already fits is + emitted byte-for-byte as before, so SQL stays readable on every engine and + under-limit output is unchanged. + +| Dialect | Budget (bytes) | Over-limit behaviour | +|---|---|---| +| Postgres | 63 | **silent truncation** | +| MySQL | 64 | error | +| Redshift | 127 | error | +| Oracle, SQL Server | 128 | error | +| Snowflake | 255 | error | +| DuckDB | 256 | error | +| BigQuery | 300 | error | +| SQLite, ClickHouse, Trino/Presto, Databricks/Spark | unbounded | — | + +The budget is one conservative number per dialect rather than a full model of +each engine's per-identifier-class rules. It is counted in **bytes**, which is +conservative for engines that count characters (MySQL, SQL Server). MySQL's +64-character *identifier* limit is used rather than its more generous 256-char +*column alias* limit, and Oracle assumes 12.2+ (128 bytes; the pre-12.2 limit +of 30 is not modelled). A dialect added without setting the field inherits the +tightest value, since over-shortening is safe and under-shortening is not. + +Two SLayer-generated identifiers colliding after shortening raises +`IdentifierCollisionError` rather than emitting ambiguous SQL. With a 32-bit +digest over the full original this is astronomically unlikely; the check exists +because a duplicate output name on a silently-truncating backend is exactly the +failure this machinery prevents. + +Join-path *table* aliases (`customers__regions`) are not yet length-bounded — +they are internal and shorter, but a sufficiently deep chain of long model +names can still collide. Tracked separately. + ## Aggregation support Most aggregations (`sum`, `avg`, `min`, `max`, `count`, `count_distinct`, diff --git a/slayer/core/errors.py b/slayer/core/errors.py index c02f38c1..c38800ef 100644 --- a/slayer/core/errors.py +++ b/slayer/core/errors.py @@ -151,6 +151,44 @@ def __init__( ) +class IdentifierCollisionError(SlayerError, ValueError): + """Raised when two distinct SLayer-generated names collapse onto one + identifier after the dialect's length fitting (DEV-1756). + + Fitting appends a 32-bit digest of the full original, so this is + astronomically unlikely — but a duplicate output name on a backend that + silently truncates is exactly the class of bug this machinery exists to + prevent, so it is raised loudly rather than left to corrupt a result set. + Also covers the case where an already-short name happens to equal another + name's fitted form, which hash width alone cannot prevent. + + Multi-inherits ``ValueError`` to match the other SLayer validation errors. + """ + + def __init__( + self, + *, + first: str, + second: str, + emitted: str, + dialect: str, + limit: int | None, + namespace: str = "identifier", + ) -> None: + self.first = first + self.second = second + self.emitted = emitted + self.dialect = dialect + self.limit = limit + self.namespace = namespace + super().__init__( + f"{namespace} collision on dialect '{dialect}' " + f"(max_identifier_bytes={limit}): {first!r} and {second!r} both " + f"emit as {emitted!r}. Rename one of the underlying models or " + f"columns to break the tie." + ) + + class ForcedFilterError(SlayerError): """Raised when the session policy's ruleset cannot be safely applied to a query. diff --git a/slayer/engine/enriched.py b/slayer/engine/enriched.py index a6ef17c1..9d72c8f5 100644 --- a/slayer/engine/enriched.py +++ b/slayer/engine/enriched.py @@ -274,6 +274,29 @@ class CrossModelMeasure(BaseModel): CrossModelMeasure.model_rebuild() +def all_projection_aliases(enriched: EnrichedQuery) -> list[str]: + """Every alias ``enriched`` can put into an emitted SELECT, in bucket order. + + DEV-1756: the superset of :func:`public_projection_aliases`, WITHOUT the + internal-prefix filtering. Hidden entries — ``_inner_*`` nested-transform + arg hoists, ``_ft*`` filter-transform extractions, ``_ts*`` change/change_pct + desugars, ORDER-BY aggregate hoists — are still projected in the inner + SELECT, so they truncate exactly like a user-declared alias and must be + length-fitted with the same map. Feeding the write pass a filtered list + would leave those references pointing at an unfitted name. + + Deduplicated (an alias can be reachable through more than one bucket) while + preserving first-seen order, so the derived rewrite map is deterministic. + """ + out: list[str] = [d.alias for d in enriched.dimensions] + out.extend(td.alias for td in enriched.time_dimensions) + out.extend(m.alias for m in enriched.measures) + out.extend(e.alias for e in enriched.expressions) + out.extend(t.alias for t in enriched.transforms) + out.extend(cm.alias for cm in enriched.cross_model_measures) + return list(dict.fromkeys(out)) + + def public_projection_aliases(enriched: EnrichedQuery) -> list[str]: """Return the ordered list of public-projection aliases for ``enriched``. diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index 8f2c0fc9..108911d1 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -20,7 +20,11 @@ from sqlglot import exp from slayer.core.enums import DEFAULT_AGGREGATIONS_BY_TYPE, DataType -from slayer.core.errors import AmbiguousModelError, ForcedFilterError +from slayer.core.errors import ( + AmbiguousModelError, + ForcedFilterError, + IdentifierCollisionError, +) from slayer.core.policy import JoinFilterRuleset, SessionPolicy from slayer.core.format import NumberFormat, NumberFormatType, format_number from slayer.core.models import ( @@ -61,6 +65,7 @@ CrossModelMeasure, EnrichedMeasure, EnrichedQuery, + all_projection_aliases, public_projection_aliases, ) from slayer.engine.enrichment import enrich_query @@ -72,10 +77,10 @@ ) from slayer.sql.client import SlayerSQLClient from slayer.sql.dialects import SqlDialect, dialect_for_ds_type, get_dialect +from slayer.sql.dialects._identifier_fit import fit_identifier from slayer.sql import engine_factory from slayer.sql.engine_factory import _runtime_fingerprint from slayer.sql.generator import SQLGenerator -from slayer.sql.reserved_keywords import SLAYER_RESERVED_KEYWORDS from slayer.sql.session_policy import ScopedTable, apply_session_policy from slayer.storage.base import StorageBackend @@ -1383,10 +1388,18 @@ async def _run_and_build( ) raise timing.record("execute", _t) - # Dialect-driven read-side decode: BigQuery reverses its alias - # mangling here so the response keys match SLayer's universal - # dotted shape. Default hook is identity for every other dialect. - rows = get_dialect(prepared.dialect).decode_result_keys(rows) + # Dialect-driven read-side decode: the base hook reverses DEV-1756 + # identifier-length fitting, and BigQuery/T-SQL additionally reverse + # their alias mangling, so response keys always match SLayer's + # universal dotted shape whatever the backend did to them. + # + # The UNFILTERED alias set is passed, not ``expected_columns``: a row + # can legitimately carry a hidden ORDER-BY hoist, and the map has to be + # able to decode it. Recomputed from the pure fitting rather than + # threaded through generation. + rows = get_dialect(prepared.dialect).decode_result_keys( + rows, aliases=all_projection_aliases(prepared.enriched), + ) columns = prepared.expected_columns if not rows else [] # [] auto-derives return SlayerResponse( data=rows, @@ -3216,11 +3229,20 @@ def _alias_to_short(alias: str) -> str: 'orders.customers.regions.name' → 'customers__regions__name' 'orders.count' → 'count' + + DEV-1756: the flattened name is then fitted to the dialect's + identifier budget. One more join hop than the example above crosses + Postgres' 63 bytes, and these names are emitted as output-column + aliases AND reused as the virtual model's ``Column.name``, so an + over-limit pair would collapse into one column. """ # Strip source model prefix stripped = alias.split(".", 1)[-1] if "." in alias else alias # Replace remaining dots with __ to encode the original join path - return stripped.replace(".", "__") + return fit_identifier( + stripped.replace(".", "__"), + limit=get_dialect(dialect).max_identifier_bytes, + ) # (inner_alias, short_name, data_type, label, description, format) column_map = [] @@ -3282,14 +3304,30 @@ def _alias_to_short(alias: str) -> str: # would either fail to parse (MySQL) or reference an alias the # mangled inner subquery doesn't expose (T-SQL). # DEV-1686: the inner ``alias`` is always dialect-quoted; the ``short`` - # output alias must also be quoted when it is a reserved word (a - # user-declared cross-model rename like ``order``, or an - # ``_alias_to_short`` that yields one), else ``AS order`` is bare and - # the wrapped SQL fails to parse/execute. + # output alias must be too. It was originally quoted only for reserved + # words, but a BARE mixed-case short is case-folded by Postgres while + # the outer stage references it quoted (DEV-1645 quotes mixed-case + # ``Column.sql`` leaves) — so any query-backed model with a mixed-case + # join path failed with ``UndefinedColumnError``. Quoting always fixes + # that and, per DEV-1756, keeps these names out of the case-folding + # namespace entirely. def _short_sql(short: str) -> str: - if short.lower() in SLAYER_RESERVED_KEYWORDS: - return exp.Identifier(this=short, quoted=True).sql(dialect=dialect) - return short + return exp.Identifier(this=short, quoted=True).sql(dialect=dialect) + + # DEV-1756: the shorts share one output-column namespace. Two that + # differ only by case would still collide were they ever emitted bare, + # and two that fit to the same string always collide, so validate the + # whole allocation before emitting. + short_owner: dict[str, str] = {} + for _, short, _, _, _, _ in column_map: + prior = short_owner.setdefault(short.casefold(), short) + if prior != short: + raise IdentifierCollisionError( + first=prior, second=short, emitted=short.casefold(), + dialect=dialect, + limit=get_dialect(dialect).max_identifier_bytes, + namespace="query-backed model column", + ) rename_parts = [ f'{exp.Identifier(this=alias, quoted=True).sql(dialect=dialect)} AS {_short_sql(short)}' @@ -3299,7 +3337,14 @@ def _short_sql(short: str) -> str: # DEV-1571 Bug 2: apply the dialect's emitted-SQL rewrite (e.g. # T-SQL bracket-mangling) so the rename clause's inner-alias # references match what the inner subquery actually projects. - wrapped_sql = get_dialect(dialect).rewrite_emitted_sql(wrapped_sql) + # DEV-1756: same alias set the inner SQL was generated with, so the + # wrapper's references land on the same fitted names. Safe to run over + # the combined string: ``generate()`` already replaced every canonical + # token inside ``inner_sql``, so this pass can only reach the wrapper's + # own references. + wrapped_sql = get_dialect(dialect).rewrite_emitted_sql( + wrapped_sql, aliases=all_projection_aliases(enriched), + ) # One Column per result column — each is potentially both a dimension # (group-by) or measure (with colon-aggregation) at query time. diff --git a/slayer/sql/dialects/_identifier_fit.py b/slayer/sql/dialects/_identifier_fit.py new file mode 100644 index 00000000..2fdac46b --- /dev/null +++ b/slayer/sql/dialects/_identifier_fit.py @@ -0,0 +1,163 @@ +"""DEV-1756: shared identifier-length fitting + the write-side substitution. + +Backends cap identifier length, and Postgres — the tightest of the Tier-1 set +at 63 bytes — **silently truncates** past it (a NOTICE, never an error). SLayer's +universal alias convention ``..`` crosses that on +a 3-hop join, so two sibling aliases collapse onto one effective output name and +the query either fails with ``AmbiguousColumnError`` or, worse, quietly returns +a column under a name nobody looks up. + +Two primitives live here: + +:func:`fit_identifier` + Shortens an over-limit identifier to ``__``. It is a PURE + function of ``name`` — the digest covers the *full original*, not the + truncated head — which is what lets the read side rebuild the + emitted->canonical map by simply re-running it, with no map threaded through + generation. Identity when the name already fits, so the overwhelmingly + common case emits byte-identical SQL. + +:func:`substitute_quoted` + Rewrites quoted identifier tokens in emitted SQL. Driven by an exact + canonical->emitted map rather than a length regex, so it can never reach + into a string literal that happens to contain a long quoted-looking span. + +Sibling of :mod:`slayer.sql.dialects._alias_mangle` (the BigQuery/T-SQL dotted +alias codec) and composes with it: those dialects size against ``encode_alias`` +via the ``expand`` hook, because their mangling *lengthens* the identifier after +fitting. +""" + +from __future__ import annotations + +import hashlib +from collections.abc import Callable, Mapping + + +#: Hex characters of digest carried in the marker. 32 bits is ample given that +#: every namespace validates its allocation and raises on collision — width only +#: affects how often that (astronomically rare) error could fire. +HASH_LEN = 8 + +#: Below this there is no room for both the marker and any readable context. +#: Every dialect SLayer supports is far above it; the guard exists so a +#: mis-configured limit fails loudly rather than emitting a useless name. +MIN_LIMIT = 16 + +#: ``_`` + digest + ``_`` +_MARKER_LEN = HASH_LEN + 2 + +#: Trimmed from the inner edges of head/tail so the marker never abuts a path +#: separator (``foo._a1b2c3d4_.bar``). +_TRIM = "._" + +#: Two-phase substitution sentinel. NUL bytes cannot appear in SQL SLayer +#: generates, so a sentinel can never be confused with real content. +_SENTINEL = "\x00\x01{}\x01\x00" + + +def _digest(name: str) -> str: + """Stable digest of the FULL original name. + + ``sha256`` rather than the builtin ``hash`` because the read side + recomputes this in a different process, where ``hash`` would be salted by + ``PYTHONHASHSEED``. Patched by tests to force collisions. + """ + return hashlib.sha256(name.encode("utf-8")).hexdigest()[:HASH_LEN] + + +def _head_bytes(name: str, n: int) -> str: + """Leading ``n`` bytes of ``name``, cut on a UTF-8 codepoint boundary.""" + if n <= 0: + return "" + return name.encode("utf-8")[:n].decode("utf-8", "ignore") + + +def _tail_bytes(name: str, n: int) -> str: + """Trailing ``n`` bytes of ``name``, cut on a UTF-8 codepoint boundary.""" + if n <= 0: + return "" + return name.encode("utf-8")[-n:].decode("utf-8", "ignore") + + +def fit_identifier( + name: str, + *, + limit: int | None, + expand: Callable[[str], str] | None = None, +) -> str: + """Shorten ``name`` to at most ``limit`` **bytes** as ``__``. + + Returns ``name`` unchanged when ``limit`` is ``None`` (unbounded dialect) or + the name already fits — shortening only kicks in when it must, so SQL stays + readable everywhere else and existing emission tests see no churn. + + Both ends are preserved because they carry the information: the head names + the root model, the tail names the actual column. In the reported repro the + two colliding aliases differ *only* in their final segment, so a head-only + truncation would render them indistinguishable in ``dry_run`` output. + + ``expand`` sizes the budget against a post-fit transform. BigQuery and T-SQL + mangle ``.`` to ``___`` *after* this runs, adding 2 bytes per dot; passing + ``encode_alias`` makes the loop shrink until the mangled form fits. The + returned value is NOT expanded — the dialect's own pass does that. + + Injective in practice, and any residual collision is caught by the caller's + per-namespace allocation check, so the scheme is collision-*detected* rather + than collision-free. + """ + grow = expand or (lambda s: s) + if limit is None or len(grow(name).encode("utf-8")) <= limit: + return name + if limit < MIN_LIMIT: + raise ValueError( + f"identifier limit must be at least MIN_LIMIT ({MIN_LIMIT}) bytes to " + f"leave room for the {_MARKER_LEN}-byte hash marker plus context; got {limit}" + ) + marker = f"_{_digest(name)}_" + # Shrink the budget until the (possibly expanded) candidate fits. The final + # iteration leaves head and tail empty, yielding the bare ``__``, + # which is <= MIN_LIMIT bytes and starts with an underscore — legal even + # unquoted, where a bare hex digest could have started with a digit. + for budget in range(limit, _MARKER_LEN - 1, -1): + avail = budget - _MARKER_LEN + tail_n = avail // 2 + head_n = avail - tail_n + head = _head_bytes(name, head_n).rstrip(_TRIM) + tail = _tail_bytes(name, tail_n).lstrip(_TRIM) + candidate = f"{head}{marker}{tail}" + if len(grow(candidate).encode("utf-8")) <= limit: + return candidate + raise ValueError( + f"cannot fit {name!r} into {limit} bytes: the supplied `expand` grows " + f"even the bare {_MARKER_LEN}-byte marker beyond the limit" + ) + + +def substitute_quoted( + sql: str, + mapping: Mapping[str, str], + *, + quote: Callable[[str], str], +) -> str: + """Replace each quoted ``canonical`` identifier token with its ``emitted`` + form, everywhere it appears. + + Two-phase (canonical -> sentinel -> emitted) so no substitution can be + re-read by a later one. With today's allocation the key set (over-limit) and + the value set (within-limit) are provably disjoint, so a single sequential + pass would also be correct; the two-phase form keeps that from becoming a + silent trap if the allocation ever changes. + + Only *quoted* occurrences move. A bare occurrence of the same text is a + different identifier — a table alias, say — and is left alone, which is what + keeps the deferred join-path-alias surface (DEV-1743) out of scope here. + """ + if not mapping: + return sql + items = sorted(mapping.items()) + for index, (canonical, _) in enumerate(items): + sql = sql.replace(quote(canonical), _SENTINEL.format(index)) + for index, (_, emitted) in enumerate(items): + sql = sql.replace(_SENTINEL.format(index), quote(emitted)) + return sql diff --git a/slayer/sql/dialects/_tier2.py b/slayer/sql/dialects/_tier2.py index c5014e69..8605ff26 100644 --- a/slayer/sql/dialects/_tier2.py +++ b/slayer/sql/dialects/_tier2.py @@ -34,6 +34,8 @@ class RedshiftDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = False + # DEV-1756: 127-byte identifier limit. + max_identifier_bytes: int | None = 127 def build_approx_count_distinct( self, @@ -52,6 +54,8 @@ class TrinoDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: Trino imposes no practical identifier-length limit. + max_identifier_bytes: int | None = None def build_approx_count_distinct( self, @@ -71,6 +75,8 @@ class PrestoDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: Presto/Athena impose no practical identifier-length limit. + max_identifier_bytes: int | None = None def build_approx_count_distinct( self, @@ -89,6 +95,8 @@ class DatabricksDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: Databricks imposes no practical identifier-length limit. + max_identifier_bytes: int | None = None def build_approx_count_distinct( self, @@ -107,6 +115,8 @@ class SparkDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: Spark imposes no practical identifier-length limit. + max_identifier_bytes: int | None = None def build_approx_count_distinct( self, @@ -127,6 +137,8 @@ class OracleDialect(SqlDialect): # the canonical 2-arg LOG(base, x) form. log10_native: bool = False log2_native: bool = False + # DEV-1756: 128 bytes on 12.2+; pre-12.2 was 30 and is not modelled. + max_identifier_bytes: int | None = 128 def build_approx_count_distinct( self, diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 7af32f60..6e721985 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -14,13 +14,15 @@ from functools import lru_cache from typing import TYPE_CHECKING, Any -from collections.abc import Callable +from collections.abc import Callable, Sequence from pydantic import BaseModel, ConfigDict from sqlglot import exp from sqlglot.dialects.dialect import Dialect as _SqlglotDialect from slayer.core.enums import TimeGranularity +from slayer.core.errors import IdentifierCollisionError +from slayer.sql.dialects._identifier_fit import fit_identifier, substitute_quoted if TYPE_CHECKING: import sqlalchemy as sa @@ -184,6 +186,16 @@ class SqlDialect(BaseModel): log10_native: bool = True log2_native: bool = True + # DEV-1756: conservative universal identifier budget, in BYTES. This is + # deliberately NOT an exact model of each backend's per-identifier-class + # rules — one number that is never too generous. Bytes are conservative for + # backends that count characters (MySQL, SQL Server). ``None`` means + # effectively unbounded, in which case every fitting hook is a no-op. + # The base default is the tightest Tier-1 value (Postgres' NAMEDATALEN-1), + # so a dialect added later without setting it over-shortens rather than + # silently truncating. + max_identifier_bytes: int | None = 63 + @property def backslash_escapes_strings(self) -> bool: """Whether this dialect's string literals treat a backslash as an escape @@ -492,41 +504,141 @@ def emit_outer_wrap( out += "\n" + offset_arg.sql(dialect=self.sqlglot_name, pretty=True) return out - def rewrite_emitted_sql(self, sql: str) -> str: - """Default: identity. Post-pass string-level rewrite of the final - generator output. + # ------------------------------------------------------------------ + # DEV-1756: identifier-length fitting + # + # Postgres truncates over-limit identifiers SILENTLY, so two long sibling + # aliases collapse onto one output name. Aliases stay canonical everywhere + # inside SLayer; they are fitted only on emission and restored on the + # result keys, so consumers never observe the dialect dependence. + # ------------------------------------------------------------------ + + def quote_identifier(self, name: str) -> str: + """``name`` wrapped in this dialect's identifier quotes.""" + return exp.Identifier(this=name, quoted=True).sql(dialect=self.sqlglot_name) + + def fit_alias(self, name: str) -> str: + """LENGTH-ONLY fitting. Identity when ``name`` already fits. + + This — not ``emit_alias`` — drives the write pass, which is why an + under-limit alias produces byte-identical SQL on every dialect, + including the ones that separately mangle dots. + """ + return fit_identifier(name, limit=self.max_identifier_bytes) + + def emit_alias(self, alias: str) -> str: + """The FINAL identifier a canonical alias reaches the SQL as. + + Equals ``fit_alias`` here; ``BigqueryDialect`` / ``TsqlDialect`` + compose their dot-mangling on top. Used to build the read-side map, so + it must match the emitted token exactly. + """ + return self.fit_alias(alias) + + def alias_rewrite_map(self, aliases: Sequence[str]) -> dict[str, str]: + """``{canonical: fitted}`` for the write pass, only where they differ. + + The collision check covers EVERY alias including the identities: an + already-short alias whose spelling equals another's fitted form is just + as much a duplicate output name, and no hash width can prevent it. + """ + if self.max_identifier_bytes is None: + return {} + allocation: dict[str, str] = {} + owner: dict[str, str] = {} + for alias in aliases: + if alias in allocation: + continue + fitted = self.fit_alias(alias) + prior = owner.get(fitted) + if prior is not None and prior != alias: + raise IdentifierCollisionError( + first=prior, second=alias, emitted=fitted, + dialect=self.sqlglot_name, limit=self.max_identifier_bytes, + namespace="projection alias", + ) + owner[fitted] = alias + allocation[alias] = fitted + return {k: v for k, v in allocation.items() if k != v} + + def decode_alias_map(self, aliases: Sequence[str]) -> dict[str, str]: + """``{emitted: canonical}`` — the read-side inverse, rebuilt by simply + re-running the (pure) fitting rather than threading a map through + generation.""" + out: dict[str, str] = {} + for alias in aliases: + emitted = self.emit_alias(alias) + if emitted != alias: + out[emitted] = alias + return out + + def _rekey_row( + self, row: dict[str, Any], mapping: dict[str, str], + ) -> dict[str, Any]: + """Apply ``mapping`` to one row's keys, refusing to let two keys + collapse onto one (which would silently drop a column's values).""" + out: dict[str, Any] = {} + for key, value in row.items(): + decoded = mapping.get(key, key) + if decoded in out: + raise IdentifierCollisionError( + first=key, second=decoded, emitted=decoded, + dialect=self.sqlglot_name, limit=self.max_identifier_bytes, + namespace="result key", + ) + out[decoded] = value + return out - Symmetric companion to ``rewrite_parsed_ast`` (the input-side - hook): write-side, applied at the end of - ``SQLGenerator.generate()`` AFTER ``_apply_outer_projection_trim``. + def rewrite_emitted_sql( + self, sql: str, *, aliases: Sequence[str] = (), + ) -> str: + """Post-pass string-level rewrite of the final generator output. + + Symmetric companion to ``rewrite_parsed_ast`` (the input-side hook): + write-side, applied at the end of ``SQLGenerator.generate()`` AFTER + ``_apply_outer_projection_trim``. + + Base impl performs the DEV-1756 length pass: each canonical alias in + ``aliases`` whose fitted form differs has its dialect-quoted token + replaced, everywhere it occurs (inner ``AS``, outer wrap projection, + ORDER BY, CTE column references). Driven by the query's own alias set + rather than a length regex, so it cannot reach into a string literal. + + ``aliases`` defaults to empty, which makes this a no-op — every caller + that does not supply an alias set keeps today's behaviour exactly. Contract: preserve query semantics. Suitable for alias renames, - identifier mangling/escape, dialect-quoting fixes. Do NOT change - query shape — use the typed ``build_*`` methods on this class for - that. + identifier mangling/escape, dialect-quoting fixes. Do NOT change query + shape — use the typed ``build_*`` methods on this class for that. - Overrides today: ``BigqueryDialect`` mangles dotted aliases that - would otherwise be rejected by BigQuery's output column-name - grammar. + Overrides today: ``BigqueryDialect`` / ``TsqlDialect`` compose their + dotted-alias mangling AFTER this length pass. """ - return sql + mapping = self.alias_rewrite_map(aliases) + if not mapping: + return sql + return substitute_quoted(sql, mapping, quote=self.quote_identifier) def decode_result_keys( self, rows: list[dict[str, Any]], + *, + aliases: Sequence[str] = (), ) -> list[dict[str, Any]]: - """Default: identity. Reverse-pass on result-row keys to undo any - write-side mangling applied by ``rewrite_emitted_sql``. + """Reverse-pass on result-row keys to undo the write-side rewrite. Called at the end of ``SlayerQueryEngine.execute()`` so consumers always see SLayer's universal alias shape (``orders._count``, - ``orders.products.category``) regardless of which dialect a - query ran on. + ``orders.products.category``) regardless of which dialect a query ran + on — and regardless of whether its aliases had to be shortened. - Overrides today: ``BigqueryDialect`` decodes the ``___`` mangling - back to dots. + Overrides today: ``BigqueryDialect`` / ``TsqlDialect`` additionally + decode the ``___`` mangling back to dots. """ - return rows + mapping = self.decode_alias_map(aliases) + if not mapping: + return rows + return [self._rekey_row(row, mapping) for row in rows] def register_udfs(self, dbapi_connection) -> None: """Default: no-op. SQLite overrides to register Python aggregate diff --git a/slayer/sql/dialects/bigquery.py b/slayer/sql/dialects/bigquery.py index 0eace4ac..33e3913d 100644 --- a/slayer/sql/dialects/bigquery.py +++ b/slayer/sql/dialects/bigquery.py @@ -28,13 +28,14 @@ import json import re from typing import TYPE_CHECKING, Any -from collections.abc import Callable +from collections.abc import Callable, Sequence import sqlalchemy as sa from sqlglot import exp from slayer.core.enums import TimeGranularity from slayer.sql.dialects._alias_mangle import decode_alias, encode_alias +from slayer.sql.dialects._identifier_fit import fit_identifier from slayer.sql.dialects.base import SqlDialect if TYPE_CHECKING: @@ -88,6 +89,8 @@ class BigqueryDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: 300-character column-name limit. + max_identifier_bytes: int | None = 300 def build_approx_count_distinct( self, @@ -130,7 +133,25 @@ def build_date_trunc( this="DATE_TRUNC", expressions=[col_expr, week_sunday], ) - def rewrite_emitted_sql(self, sql: str) -> str: + def fit_alias(self, name: str) -> str: + """DEV-1756: size the length budget against the POST-mangle form. + + ``rewrite_emitted_sql`` expands every ``.`` to ``___`` after fitting, + adding 2 bytes per dot, so fitting to the raw limit would bust it on a + deep chain. The returned value is still dotted — the regex below does + the mangling. + """ + return fit_identifier( + name, limit=self.max_identifier_bytes, expand=encode_alias, + ) + + def emit_alias(self, alias: str) -> str: + """The final identifier: length-fitted, then dot-mangled.""" + return encode_alias(self.fit_alias(alias)) + + def rewrite_emitted_sql( + self, sql: str, *, aliases: Sequence[str] = (), + ) -> str: """Replace ``.`` with ``___`` inside backtick-quoted identifiers. Applied as a post-pass on the BigQuery dialect's final SQL so @@ -138,7 +159,17 @@ def rewrite_emitted_sql(self, sql: str) -> str: references to those aliases (``ORDER BY \\`orders._count\\``) comply with BigQuery's column-name grammar. + + DEV-1756: the base class's LENGTH pass runs first. An under-limit alias + is untouched by it (``fit_alias`` is the identity), so the regex below + sees exactly what it sees today and the output stays byte-identical. + An over-limit alias is rewritten to ``__`` whose + head/tail are still dotted, so the regex mangles it here — yielding + ``encode_alias(fit_alias(a))``, which is what ``emit_alias`` returns. + Because the fitted form is mangled by this same pass rather than + arriving pre-mangled, there is no double-encoding. """ + sql = super().rewrite_emitted_sql(sql, aliases=aliases) return _DOTTED_ALIAS_RE.sub( lambda m: f"`{encode_alias(m.group(1))}`", sql ) @@ -146,11 +177,26 @@ def rewrite_emitted_sql(self, sql: str) -> str: def decode_result_keys( self, rows: list[dict[str, Any]], + *, + aliases: Sequence[str] = (), ) -> list[dict[str, Any]]: """Reverse the BigQuery alias mangling on result-row keys so consumers see SLayer's universal dotted alias shape regardless of - whether the query ran against BigQuery or another dialect.""" - return [{decode_alias(k): v for k, v in row.items()} for row in rows] + whether the query ran against BigQuery or another dialect. + + DEV-1756: keys produced by a length-fitted alias are not recoverable + from the key alone, so the ``emitted -> canonical`` map is consulted + first; anything outside it falls back to the pure ``___`` -> ``.`` + bijection, preserving today's behaviour for short aliases. + """ + mapping = self.decode_alias_map(aliases) + return [ + self._rekey_row( + {decode_alias(k) if k not in mapping else k: v for k, v in row.items()}, + mapping, + ) + for row in rows + ] def build_engine( self, diff --git a/slayer/sql/dialects/clickhouse.py b/slayer/sql/dialects/clickhouse.py index 51265822..7eab146c 100644 --- a/slayer/sql/dialects/clickhouse.py +++ b/slayer/sql/dialects/clickhouse.py @@ -21,6 +21,8 @@ class ClickhouseDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: ClickHouse imposes no practical identifier-length limit. + max_identifier_bytes: int | None = None def build_median( self, diff --git a/slayer/sql/dialects/duckdb.py b/slayer/sql/dialects/duckdb.py index c8a9e7da..8b4eba52 100644 --- a/slayer/sql/dialects/duckdb.py +++ b/slayer/sql/dialects/duckdb.py @@ -21,6 +21,8 @@ class DuckdbDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: DuckDB accepts long identifiers; 256 is a safe documented ceiling. + max_identifier_bytes: int | None = 256 def build_approx_count_distinct( self, diff --git a/slayer/sql/dialects/mysql.py b/slayer/sql/dialects/mysql.py index c9fd6974..c861d433 100644 --- a/slayer/sql/dialects/mysql.py +++ b/slayer/sql/dialects/mysql.py @@ -25,6 +25,8 @@ class MysqlDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: 64-char identifier limit. MySQL allows 256 for column ALIASES specifically, but one conservative number keeps the rule simple, and MySQL errors rather than truncating. + max_identifier_bytes: int | None = 64 def build_median( self, diff --git a/slayer/sql/dialects/postgres.py b/slayer/sql/dialects/postgres.py index 65909149..4868d7e8 100644 --- a/slayer/sql/dialects/postgres.py +++ b/slayer/sql/dialects/postgres.py @@ -45,6 +45,8 @@ class PostgresDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: NAMEDATALEN 64 -> 63 usable bytes; over-length names are SILENTLY truncated. + max_identifier_bytes: int | None = 63 def rewrite_target_ast(self, tree: exp.Expression) -> exp.Expression: """DEV-1576: numeric-cast the first arg of every 2-arg ROUND so diff --git a/slayer/sql/dialects/snowflake.py b/slayer/sql/dialects/snowflake.py index 2fe7a87a..fbe89f8b 100644 --- a/slayer/sql/dialects/snowflake.py +++ b/slayer/sql/dialects/snowflake.py @@ -193,6 +193,8 @@ class so ``engine_factory`` / ``client`` stay dialect-agnostic. log10_native: bool = True # No native LOG2 — falls through to canonical ``LOG(2, x)`` form. log2_native: bool = False + # DEV-1756: 255-character identifier limit. + max_identifier_bytes: int | None = 255 def build_approx_count_distinct( self, diff --git a/slayer/sql/dialects/sqlite.py b/slayer/sql/dialects/sqlite.py index 13f2994d..9b66059c 100644 --- a/slayer/sql/dialects/sqlite.py +++ b/slayer/sql/dialects/sqlite.py @@ -406,6 +406,8 @@ class SqliteDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True + # DEV-1756: SQLite imposes no identifier-length limit. + max_identifier_bytes: int | None = None def build_date_trunc( self, diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index 59193fcb..11c7b5a6 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -28,13 +28,14 @@ import re from typing import Any -from collections.abc import Callable +from collections.abc import Callable, Sequence import sqlglot from sqlglot import exp from slayer.core.enums import TimeGranularity from slayer.sql.dialects._alias_mangle import decode_alias, encode_alias +from slayer.sql.dialects._identifier_fit import fit_identifier from slayer.sql.dialects.base import SqlDialect, _build_covar_decomposition @@ -70,6 +71,8 @@ class TsqlDialect(SqlDialect): explain_postfix: str = "; SET SHOWPLAN_ALL OFF" log10_native: bool = True log2_native: bool = False + # DEV-1756: sysname is nvarchar(128), i.e. 128 characters. + max_identifier_bytes: int | None = 128 def build_approx_count_distinct( self, @@ -334,7 +337,24 @@ def emit_outer_wrap( # DEV-1571 Bug 2: bracketed dotted-alias mangling # ------------------------------------------------------------------ - def rewrite_emitted_sql(self, sql: str) -> str: + def fit_alias(self, name: str) -> str: + """DEV-1756: size the length budget against the POST-mangle form. + + ``rewrite_emitted_sql`` expands every ``.`` to ``___`` after fitting, + adding 2 bytes per dot, so fitting to the raw 128-byte limit would bust + it on a deep chain. The returned value is still dotted. + """ + return fit_identifier( + name, limit=self.max_identifier_bytes, expand=encode_alias, + ) + + def emit_alias(self, alias: str) -> str: + """The final identifier: length-fitted, then dot-mangled.""" + return encode_alias(self.fit_alias(alias)) + + def rewrite_emitted_sql( + self, sql: str, *, aliases: Sequence[str] = (), + ) -> str: """Replace ``.`` with ``___`` inside bracket-quoted identifiers. T-SQL's ``ORDER BY`` resolver does not treat ``[a.b]`` as a @@ -348,7 +368,15 @@ def rewrite_emitted_sql(self, sql: str) -> str: Uses the same bijection as ``BigqueryDialect`` (shared encode in ``slayer.sql.dialects._alias_mangle``); only the regex anchor differs. + + DEV-1756: the base class's LENGTH pass runs first. An under-limit alias + is untouched by it (``fit_alias`` is the identity), so the regex below + sees exactly what it sees today and the output stays byte-identical. + An over-limit alias arrives as ``__`` with head/tail + still dotted, so this pass mangles it — yielding what ``emit_alias`` + returns, with no double-encoding. """ + sql = super().rewrite_emitted_sql(sql, aliases=aliases) return _TSQL_DOTTED_ALIAS_RE.sub( lambda m: f"[{encode_alias(m.group(1))}]", sql ) @@ -356,9 +384,23 @@ def rewrite_emitted_sql(self, sql: str) -> str: def decode_result_keys( self, rows: list[dict[str, Any]], + *, + aliases: Sequence[str] = (), ) -> list[dict[str, Any]]: """Reverse the T-SQL alias mangling on result-row keys so consumers see SLayer's universal dotted alias shape regardless of whether the query ran against T-SQL or another dialect. + + DEV-1756: keys produced by a length-fitted alias are not recoverable + from the key alone, so the ``emitted -> canonical`` map is consulted + first; anything outside it falls back to the pure ``___`` -> ``.`` + bijection, preserving today's behaviour for short aliases. """ - return [{decode_alias(k): v for k, v in row.items()} for row in rows] + mapping = self.decode_alias_map(aliases) + return [ + self._rekey_row( + {decode_alias(k) if k not in mapping else k: v for k, v in row.items()}, + mapping, + ) + for row in rows + ] diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 82dab6b4..a0a6a135 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -19,9 +19,15 @@ DataType, TimeGranularity, ) -from slayer.core.errors import UnresolvableOrderColumnError -from slayer.engine.enriched import EnrichedMeasure, EnrichedQuery, public_projection_aliases +from slayer.core.errors import IdentifierCollisionError, UnresolvableOrderColumnError +from slayer.engine.enriched import ( + EnrichedMeasure, + EnrichedQuery, + all_projection_aliases, + public_projection_aliases, +) from slayer.sql.dialects import SqlDialect, get_dialect +from slayer.sql.dialects._identifier_fit import fit_identifier from slayer.sql.reserved_keywords import ( SLAYER_RESERVED_KEYWORDS, prequote_reserved_identifiers, @@ -259,17 +265,26 @@ def _parse_window_duration(value: str) -> list[tuple[int, str]]: return parts -def _cte_name_from_alias(prefix: str, alias: str) -> str: +def _cte_name_from_alias(prefix: str, alias: str, *, limit: int | None = None) -> str: """Build a unique CTE name from a measure alias. Dots are replaced with ``__`` (double underscore) to avoid collision with aliases that already contain underscores. E.g.: - ``orders.revenue_sum`` -> ``_fm_orders__revenue_sum`` - ``orders_v2.revenue_sum`` -> ``_fm_orders_v2__revenue_sum`` + + DEV-1756: ``limit`` is the dialect's ``max_identifier_bytes``. The whole + result is fitted, PREFIX INCLUDED — a ``cp_value_12_`` prefix eats 12 of + Postgres' 63 bytes before the alias is even considered. CTE names are + emitted unquoted, so an over-limit definition and its reference would be + truncated to the same thing by the server today (harmless) or to a + collision with a sibling CTE (not harmless). Stays a pure function of + ``(prefix, alias, limit)``, so every call site derives the same name and + definition and reference cannot drift. """ sanitized = alias.replace(".", "__") sanitized = re.sub(r"[^a-zA-Z0-9_]", "_", sanitized) - return prefix + sanitized + return fit_identifier(prefix + sanitized, limit=limit) def _alias_prefixes(model_name: str) -> list: @@ -403,6 +418,31 @@ def __init__(self, dialect: str | SqlDialect = "postgres"): self._dialect: SqlDialect = dialect else: self._dialect = get_dialect(dialect) + # DEV-1756: CTE-name allocation for the statement being generated, + # ``emitted -> (prefix, alias)``. Reset per ``generate()``. + self._cte_names: dict[str, tuple[str, str]] = {} + + def _cte_name(self, prefix: str, alias: str) -> str: + """Allocate a length-fitted CTE name, refusing a collision. + + DEV-1756: CTE names live in one namespace per statement and are emitted + UNQUOTED, so two that fit to the same string would produce a query that + silently references the wrong CTE. Repeat calls with the same + ``(prefix, alias)`` are the same name, not a collision — several code + paths re-derive a CTE's name to reference it. + """ + name = _cte_name_from_alias( + prefix, alias, limit=self._dialect.max_identifier_bytes, + ) + owner = (prefix, alias) + prior = self._cte_names.setdefault(name, owner) + if prior != owner: + raise IdentifierCollisionError( + first=f"{prior[0]}{prior[1]}", second=f"{prefix}{alias}", + emitted=name, dialect=self.dialect, + limit=self._dialect.max_identifier_bytes, namespace="CTE name", + ) + return name @property def dialect(self) -> str: @@ -596,6 +636,8 @@ def generate( raise ValueError( f"render_mode must be 'outer' or 'wrapped', got {render_mode!r}" ) + # DEV-1756: CTE names are allocated per statement. + self._cte_names = {} has_isolated = any(_has_cross_model_filter(m) for m in enriched.measures) has_windowed = any(_is_windowed_measure(m) for m in enriched.measures) has_cross_model = bool(enriched.cross_model_measures) @@ -626,12 +668,19 @@ def generate( if render_mode == "outer": sql = self._apply_outer_projection_trim(sql=sql, enriched=enriched) - # Dialect-driven post-pass: BigQuery mangles dotted aliases here. - # Default hook is identity for every other dialect (Postgres-shaped - # SqlDialect base). Fires for BOTH render modes — inner CTE column - # names are subject to the same dialect alias rules as the outer - # projection. - sql = self._dialect.rewrite_emitted_sql(sql) + # Dialect-driven post-pass: the base class fits over-limit projection + # aliases to the dialect's identifier budget (DEV-1756); BigQuery and + # T-SQL additionally mangle dotted aliases. Fires for BOTH render modes + # — inner CTE column names are subject to the same dialect alias rules + # as the outer projection. + # + # The alias set is the UNFILTERED one: hidden ORDER-BY hoists and + # ``_inner_*`` / ``_ft*`` / ``_ts*`` entries are projected in the inner + # SELECT and truncate exactly like user-declared aliases, so a filtered + # list would leave their references pointing at an unfitted name. + sql = self._dialect.rewrite_emitted_sql( + sql, aliases=all_projection_aliases(enriched), + ) return sql def _apply_outer_projection_trim( @@ -753,7 +802,7 @@ def _build_combined(self, enriched: EnrichedQuery, # --- Cross-model measure CTEs --- seen_cm_ctes: set = set() for cm in enriched.cross_model_measures: - cte_name = _cte_name_from_alias("_cm_", cm.alias) + cte_name = self._cte_name("_cm_", cm.alias) if cte_name in seen_cm_ctes: measure_cte_refs.append((cte_name, cm.alias, None)) continue @@ -838,7 +887,7 @@ def _build_combined(self, enriched: EnrichedQuery, for measure in enriched.measures: if not _is_windowed_measure(measure): continue - cte_name = _cte_name_from_alias("_wm_", measure.alias) + cte_name = self._cte_name("_wm_", measure.alias) ctes.append((cte_name, self._generate_window_measure_cte(enriched=enriched, measure=measure))) measure_cte_refs.append((cte_name, measure.alias, None)) @@ -846,7 +895,7 @@ def _build_combined(self, enriched: EnrichedQuery, for measure in enriched.measures: if not _has_cross_model_filter(measure): continue - cte_name = _cte_name_from_alias("_fm_", measure.alias) + cte_name = self._cte_name("_fm_", measure.alias) # Measure aggregation without CASE WHEN (the join IS the filter) unfiltered = copy.copy(measure) @@ -1772,9 +1821,9 @@ def _build_consecutive_periods_ctes( layer_num: int, ) -> tuple[list[tuple[str, str]], list[tuple[str, str]]]: partition_aliases = getattr(transform, "partition_aliases", []) or [] - reset_alias = _cte_name_from_alias("_cp_reset_", transform.alias) - reset_cte = _cte_name_from_alias(f"cp_reset_{layer_num}_", transform.alias) - value_cte = _cte_name_from_alias(f"cp_value_{layer_num}_", transform.alias) + reset_alias = self._cte_name("_cp_reset_", transform.alias) + reset_cte = self._cte_name(f"cp_reset_{layer_num}_", transform.alias) + value_cte = self._cte_name(f"cp_value_{layer_num}_", transform.alias) def _quoted_col(name: str) -> exp.Column: return exp.Column(this=exp.to_identifier(name, quoted=True)) diff --git a/tests/dialects/test_bigquery.py b/tests/dialects/test_bigquery.py index fade31de..210b1db2 100644 --- a/tests/dialects/test_bigquery.py +++ b/tests/dialects/test_bigquery.py @@ -438,7 +438,7 @@ async def test_generator_dispatches_through_rewrite_emitted_sql_hook() -> None: type(gen._dialect), "rewrite_emitted_sql", autospec=True, - side_effect=lambda self, sql: sql, + side_effect=lambda self, sql, **kw: sql, ) as spy: gen.generate(enriched=enriched) assert spy.called, ( @@ -480,7 +480,7 @@ async def test_engine_dispatches_through_decode_result_keys_hook() -> None: PostgresDialect, "decode_result_keys", autospec=True, - side_effect=lambda self, rows: rows, + side_effect=lambda self, rows, **kw: rows, ) as spy: await engine.execute(SlayerQuery( source_model="orders", diff --git a/tests/dialects/test_identifier_fit.py b/tests/dialects/test_identifier_fit.py new file mode 100644 index 00000000..8b70d707 --- /dev/null +++ b/tests/dialects/test_identifier_fit.py @@ -0,0 +1,502 @@ +"""DEV-1756: the identifier-length primitive and its dialect wiring. + +Postgres' NAMEDATALEN is 64, so identifiers are capped at 63 BYTES and anything +longer is SILENTLY truncated (a NOTICE, never an error). SLayer's projection +aliases (``..``) cross that on a 3-hop join, so two +sibling aliases can collapse onto one effective output name. + +``fit_identifier`` is the shared primitive that shortens an over-limit +identifier to a deterministic ``__``. It is a PURE function +of ``name`` (the digest covers the full original), which is what lets the read +side rebuild the emitted->canonical map without threading anything through +generation. + +``substitute_quoted`` is the write-side primitive: a TWO-PHASE replacement +(canonical -> sentinel -> final) so no substitution can be re-read by a later +one. Today the key set (over-limit) and the value set (within-limit) are +provably disjoint, so a naive sequential replace would also be correct — the +two-phase form is defence against that invariant being weakened later, and is +tested directly rather than through the alias API where the disjointness makes +a cascade unconstructible. + +Emission/behaviour tests for the three surfaces live in +``tests/test_dev1756_identifier_length.py``; live execution is in the Postgres +integration suite. +""" + +from __future__ import annotations + +import hashlib +import os +import re +import subprocess +import sys + +import pytest + +from slayer.core.errors import IdentifierCollisionError +from slayer.sql.dialects import _ALL_DIALECTS, get_dialect +from slayer.sql.dialects._alias_mangle import encode_alias + +# The feature under test. +from slayer.sql.dialects._identifier_fit import ( + HASH_LEN, + MIN_LIMIT, + fit_identifier, + substitute_quoted, +) + + +# The DEV-1756 repro pair: 73 and 74 bytes, sharing a 63-byte prefix. +LONG_NAME = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name" +LONG_EMAIL = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email" + +# Two over-limit names that differ ONLY in the middle, so head and tail both +# survive fitting identically — the shape needed to force a digest collision. +TWIN_A = "SandboxAlpha." * 3 + "111" + ".SandboxOmega" * 3 +TWIN_B = "SandboxAlpha." * 3 + "222" + ".SandboxOmega" * 3 + +# Every limit SLayer configures on a dialect, plus Postgres' binding 63. +ALL_LIMITS = (63, 64, 127, 128, 255, 256, 300) + +_UNQUOTED_IDENT_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +_MARKER_RE = re.compile(rf"_([0-9a-f]{{{HASH_LEN}}})_") + + +def _nbytes(s: str) -> int: + return len(s.encode("utf-8")) + + +def _dq(name: str) -> str: + """ANSI double-quote, the shape ``substitute_quoted`` is handed.""" + return f'"{name}"' + + +# --------------------------------------------------------------------------- +# fit_identifier — core contract +# --------------------------------------------------------------------------- + + +class TestFitIdentifierCore: + def test_repro_pair_is_actually_over_the_postgres_limit(self) -> None: + """Guard the premise: without this the rest of the file is vacuous.""" + assert _nbytes(LONG_NAME) == 73 + assert _nbytes(LONG_EMAIL) == 74 + assert LONG_NAME.encode()[:63] == LONG_EMAIL.encode()[:63] + + def test_under_limit_returned_unchanged(self) -> None: + """The common path must be a true identity — no hash, no allocation.""" + assert fit_identifier("orders.revenue_sum", limit=63) == "orders.revenue_sum" + + def test_exactly_at_limit_returned_unchanged(self) -> None: + name = "a" * 63 + assert fit_identifier(name, limit=63) == name + + def test_one_byte_over_limit_is_shortened(self) -> None: + name = "a" * 64 + assert fit_identifier(name, limit=63) != name + + def test_none_limit_is_a_no_op(self) -> None: + """Unbounded dialects (SQLite/ClickHouse/Trino/...) never shorten.""" + assert fit_identifier(LONG_EMAIL, limit=None) == LONG_EMAIL + + @pytest.mark.parametrize("limit", ALL_LIMITS) + def test_output_within_limit_bytes(self, limit: int) -> None: + name = "Sandbox" * 80 # 560 bytes — over every configured limit + assert _nbytes(fit_identifier(name, limit=limit)) <= limit + + def test_deterministic_across_calls(self) -> None: + assert fit_identifier(LONG_EMAIL, limit=63) == fit_identifier(LONG_EMAIL, limit=63) + + def test_marker_is_exactly_sha256_of_the_full_original(self) -> None: + """Pin the digest's ALGORITHM, POSITION and INPUT — not merely that + eight hex characters appear somewhere.""" + got = fit_identifier(LONG_EMAIL, limit=63) + match = _MARKER_RE.search(got) + assert match, got + expected = hashlib.sha256(LONG_EMAIL.encode()).hexdigest()[:HASH_LEN] + assert match.group(1) == expected + + def test_digest_is_not_process_dependent(self) -> None: + """The read side recomputes the map in a DIFFERENT process, so the + digest must not depend on PYTHONHASHSEED (i.e. not builtin ``hash``).""" + env = os.environ.copy() + env["PYTHONHASHSEED"] = "12345" + out = subprocess.run( + [ + sys.executable, "-c", + "from slayer.sql.dialects._identifier_fit import fit_identifier;" + f"print(fit_identifier({LONG_EMAIL!r}, limit=63))", + ], + capture_output=True, text=True, check=True, env=env, + ) + assert out.stdout.strip() == fit_identifier(LONG_EMAIL, limit=63) + + def test_head_and_tail_both_preserved(self) -> None: + """Readability contract: the root model AND the leaf column survive, + which is what makes colliding siblings tellable apart in dry_run SQL.""" + got = fit_identifier(LONG_EMAIL, limit=63) + assert got.startswith("SandboxInvoiceV2") + assert got.endswith("email") + + def test_repro_siblings_differ_outside_the_hash(self) -> None: + """The two aliases that collide on Postgres must stay distinguishable + by eye, not only by digest.""" + a = fit_identifier(LONG_NAME, limit=63) + b = fit_identifier(LONG_EMAIL, limit=63) + assert a != b + assert a.endswith("name") and b.endswith("email") + + def test_distinct_inputs_sharing_63_byte_prefix_produce_distinct_outputs(self) -> None: + a = fit_identifier(LONG_NAME, limit=63) + b = fit_identifier(LONG_EMAIL, limit=63) + assert a.encode()[:63] != b.encode()[:63] + + def test_shape_is_head_underscore_hash_underscore_tail(self) -> None: + got = fit_identifier(LONG_EMAIL, limit=63) + assert _MARKER_RE.search(got), got + + def test_separators_are_trimmed_next_to_the_marker(self) -> None: + """``head.rstrip("._")`` / ``tail.lstrip("._")`` — otherwise a budget + cut landing on a path separator yields ``foo._a1b2c3d4_.bar``.""" + # 'a.' repeated: every even byte offset lands on a '.', so an untrimmed + # head/tail would abut the marker with a separator. + name = "a." * 60 + got = fit_identifier(name, limit=63) + match = _MARKER_RE.search(got) + assert match, got + head, tail = got[: match.start()], got[match.end():] + assert not head.endswith((".", "_")), got + assert not tail.startswith((".", "_")), got + + def test_minimum_budget_form_is_legal(self) -> None: + """At the tightest legal budget the head/tail collapse away and the + result is the bare ``__`` marker — which must still be a legal + leading character (a bare hex digest can start with a digit).""" + got = fit_identifier(LONG_EMAIL, limit=MIN_LIMIT) + assert _nbytes(got) <= MIN_LIMIT + assert _UNQUOTED_IDENT_RE.match(got.replace(".", "_")), got + + +# --------------------------------------------------------------------------- +# fit_identifier — edge cases +# --------------------------------------------------------------------------- + + +class TestFitIdentifierEdges: + def test_multibyte_never_splits_a_codepoint(self) -> None: + name = "é" * 60 # 120 bytes of 2-byte codepoints + got = fit_identifier(name, limit=63) + got.encode("utf-8").decode("utf-8") # would raise if a codepoint split + assert _nbytes(got) <= 63 + + def test_multibyte_bound_is_bytes_not_characters(self) -> None: + name = "é" * 60 + got = fit_identifier(name, limit=63) + assert _nbytes(got) <= 63 < len(name) * 2 + + def test_unquoted_legality_preserved_for_flat_input(self) -> None: + """Surfaces 3 (CTE names) and 4 (virtual-model shorts) are emitted + UNQUOTED, so a fitted flat name must stay a legal bare identifier.""" + flat = "_cm_" + "SandboxSubscription__SandboxCustomer__SandboxConsumer__" * 2 + got = fit_identifier(flat, limit=63) + assert _UNQUOTED_IDENT_RE.match(got), got + + @pytest.mark.parametrize("limit", range(MIN_LIMIT, 40)) + def test_never_starts_with_a_digit(self, limit: int) -> None: + """A bare hex digest can begin with a digit, which is illegal unquoted + on several dialects.""" + assert not fit_identifier(LONG_EMAIL, limit=limit)[0].isdigit() + + @pytest.mark.parametrize("limit", range(MIN_LIMIT, 40)) + def test_tiny_budget_still_within_limit(self, limit: int) -> None: + assert _nbytes(fit_identifier(LONG_EMAIL, limit=limit)) <= limit + + def test_head_trimmed_to_empty_still_legal(self) -> None: + """A name whose head budget lands entirely inside separators must not + yield a leading-separator-then-digit mess.""" + name = "." * 40 + "abcdefghij" * 5 + got = fit_identifier(name, limit=MIN_LIMIT) + assert _nbytes(got) <= MIN_LIMIT + assert not got[0].isdigit() + + def test_punctuation_heavy_name(self) -> None: + got = fit_identifier("a._." * 40, limit=63) + assert _nbytes(got) <= 63 + + def test_limit_below_minimum_raises(self) -> None: + with pytest.raises(ValueError, match="MIN_LIMIT|too small|at least"): + fit_identifier(LONG_EMAIL, limit=MIN_LIMIT - 1) + + +# --------------------------------------------------------------------------- +# fit_identifier — the `expand` hook (BigQuery / T-SQL dot-mangling) +# --------------------------------------------------------------------------- + + +class TestFitIdentifierExpand: + def test_post_mangle_length_within_limit(self) -> None: + """BigQuery/T-SQL mangle `.` -> `___` AFTER fitting, adding 2 bytes per + dot. Fitting to the raw limit would then bust it, so the budget must be + computed against the expanded form.""" + name = ".".join(["Sandbox" * 4] * 6) # many dots, well over 128 + got = fit_identifier(name, limit=128, expand=encode_alias) + assert _nbytes(encode_alias(got)) <= 128 + + def test_expand_is_not_applied_to_the_return_value(self) -> None: + """``fit_identifier`` only SIZES against the expansion; the dialect's + own regex performs the actual mangling.""" + name = ".".join(["Sandbox" * 4] * 6) + got = fit_identifier(name, limit=128, expand=encode_alias) + assert "___" not in got + + def test_expand_under_limit_is_identity(self) -> None: + assert fit_identifier("a.b", limit=128, expand=encode_alias) == "a.b" + + def test_aggressively_expanding_transform_still_fits(self) -> None: + """The budget loop must keep shrinking until the EXPANDED form fits, + even when the expansion is far more aggressive than dot-mangling.""" + def explode(s: str) -> str: + return s + "#" * (40 * s.count(".")) + + name = "a.b.c" * 20 + got = fit_identifier(name, limit=63, expand=explode) + assert _nbytes(explode(got)) <= 63 + + +# --------------------------------------------------------------------------- +# substitute_quoted — the write-side primitive (two-phase) +# --------------------------------------------------------------------------- + + +class TestSubstituteQuoted: + def test_empty_mapping_is_identity(self) -> None: + sql = 'SELECT 1 AS "a.b"' + assert substitute_quoted(sql, {}, quote=_dq) == sql + + def test_replaces_every_occurrence(self) -> None: + sql = 'SELECT "a.b" FROM (SELECT x AS "a.b") AS _o ORDER BY "a.b"' + got = substitute_quoted(sql, {"a.b": "z"}, quote=_dq) + assert got.count('"z"') == 3 + assert '"a.b"' not in got + + def test_chained_mapping_does_not_cascade(self) -> None: + """THE two-phase requirement: with ``A -> B`` and ``B -> C`` in one + map, the ``A`` occurrence must land on ``B`` and STOP. A naive + sequential ``str.replace`` would carry it on to ``C``.""" + sql = 'SELECT "A", "B"' + got = substitute_quoted(sql, {"A": "B", "B": "C"}, quote=_dq) + assert got == 'SELECT "B", "C"' + + def test_chained_mapping_is_order_independent(self) -> None: + sql = 'SELECT "A", "B"' + forward = substitute_quoted(sql, {"A": "B", "B": "C"}, quote=_dq) + reverse = substitute_quoted(sql, {"B": "C", "A": "B"}, quote=_dq) + assert forward == reverse == 'SELECT "B", "C"' + + def test_swap_is_not_a_cascade(self) -> None: + """``A -> B`` and ``B -> A`` simultaneously — only a two-phase pass + gets this right.""" + got = substitute_quoted('SELECT "A", "B"', {"A": "B", "B": "A"}, quote=_dq) + assert got == 'SELECT "B", "A"' + + def test_only_quoted_occurrences_are_replaced(self) -> None: + """A bare (unquoted) occurrence of the same text is a different + identifier — a table alias, say — and must be left alone.""" + sql = 'SELECT tbl.col AS "tbl.col" FROM t' + got = substitute_quoted(sql, {"tbl.col": "z"}, quote=_dq) + assert got == 'SELECT tbl.col AS "z" FROM t' + + def test_does_not_reach_into_string_literals(self) -> None: + """The pass is keyed on exact quoted tokens, never on a length regex, + so a long run of text between two double quotes inside a literal is + untouched.""" + literal = "x" * 90 + sql = f'SELECT 1 AS "a.b" WHERE note LIKE \'%"{literal}"%\'' + got = substitute_quoted(sql, {"a.b": "z"}, quote=_dq) + assert f'"{literal}"' in got + + def test_sentinel_cannot_leak_into_the_output(self) -> None: + """Whatever sentinel the two-phase pass uses must not survive, even if + the SQL happens to contain sentinel-looking text.""" + sql = 'SELECT "a.b", \'\\x00 0 \\x00\' AS lit' + got = substitute_quoted(sql, {"a.b": "z"}, quote=_dq) + assert "\x00" not in got.replace("\\x00", "") + + +# --------------------------------------------------------------------------- +# Dialect wiring +# --------------------------------------------------------------------------- + + +# Conservative universal byte budgets. NOT an exact model of each backend's +# per-identifier-class rules: MySQL's 64-char *identifier* limit is used rather +# than its 256-char *column-alias* limit, and Oracle assumes 12.2+ (128, not the +# pre-12.2 30). Bytes are conservative for char-counting backends. +EXPECTED_LIMITS = { + "postgres": 63, + "mysql": 64, + "redshift": 127, + "oracle": 128, + "tsql": 128, + "snowflake": 255, + "duckdb": 256, + "bigquery": 300, + "sqlite": None, + "clickhouse": None, + "trino": None, + "presto": None, + "databricks": None, + "spark": None, +} + + +class TestDialectLimits: + def test_every_registered_dialect_is_covered(self) -> None: + assert {d.sqlglot_name for d in _ALL_DIALECTS} == set(EXPECTED_LIMITS) + + @pytest.mark.parametrize("name,expected", sorted(EXPECTED_LIMITS.items(), key=lambda kv: kv[0])) + def test_configured_limit(self, name: str, expected: int | None) -> None: + assert get_dialect(name).max_identifier_bytes == expected + + def test_base_default_is_conservative(self) -> None: + """A future dialect that forgets to set the field must inherit the + TIGHTEST limit, not an unbounded one — over-shortening is safe.""" + from slayer.sql.dialects.base import SqlDialect + + assert SqlDialect().max_identifier_bytes == 63 + + @pytest.mark.parametrize("name", sorted(EXPECTED_LIMITS)) + def test_emit_alias_identity_under_limit(self, name: str) -> None: + """Only BigQuery/T-SQL transform a short alias (dot-mangling); every + other dialect must leave it byte-identical.""" + got = get_dialect(name).emit_alias("orders.revenue_sum") + if name in ("bigquery", "tsql"): + assert got == encode_alias("orders.revenue_sum") + else: + assert got == "orders.revenue_sum" + + @pytest.mark.parametrize("name", sorted(EXPECTED_LIMITS)) + def test_fit_alias_identity_under_limit(self, name: str) -> None: + """``fit_alias`` is the LENGTH-ONLY half — identity on every dialect + for a short alias, which is what makes the write pass a no-op.""" + assert get_dialect(name).fit_alias("orders.revenue_sum") == "orders.revenue_sum" + + @pytest.mark.parametrize("name", ["sqlite", "clickhouse", "trino", "presto", "databricks", "spark"]) + def test_unbounded_dialects_never_shorten(self, name: str) -> None: + assert get_dialect(name).emit_alias(LONG_EMAIL) == LONG_EMAIL + + def test_postgres_shortens_over_limit(self) -> None: + got = get_dialect("postgres").emit_alias(LONG_EMAIL) + assert got != LONG_EMAIL + assert _nbytes(got) <= 63 + + def test_bigquery_emit_alias_is_mangled_and_fitted(self) -> None: + """BigQuery's ``emit_alias`` must be the FINAL identifier reaching the + SQL: length-fitted first, then dot-mangled.""" + long_dotted = ".".join(["Sandbox" * 6] * 8) # way over 300 + bq = get_dialect("bigquery") + got = bq.emit_alias(long_dotted) + assert "." not in got + assert _nbytes(got) <= 300 + assert got == encode_alias(bq.fit_alias(long_dotted)) + + def test_tsql_emit_alias_is_mangled_and_fitted(self) -> None: + long_dotted = ".".join(["Sandbox" * 4] * 8) # over 128 + tsql = get_dialect("tsql") + got = tsql.emit_alias(long_dotted) + assert "." not in got + assert _nbytes(got) <= 128 + assert got == encode_alias(tsql.fit_alias(long_dotted)) + + +# --------------------------------------------------------------------------- +# alias_rewrite_map — the collision guard +# --------------------------------------------------------------------------- + + +class TestAliasRewriteMap: + def test_returns_only_differing_entries(self) -> None: + pg = get_dialect("postgres") + got = pg.alias_rewrite_map(["orders.status", LONG_EMAIL]) + assert "orders.status" not in got + assert got[LONG_EMAIL] == pg.fit_alias(LONG_EMAIL) + + def test_empty_when_nothing_over_limit(self) -> None: + assert get_dialect("postgres").alias_rewrite_map(["a.b", "c.d"]) == {} + + def test_empty_alias_list(self) -> None: + assert get_dialect("postgres").alias_rewrite_map([]) == {} + + def test_unbounded_dialect_map_is_empty(self) -> None: + assert get_dialect("sqlite").alias_rewrite_map([LONG_NAME, LONG_EMAIL]) == {} + + def test_duplicate_canonical_aliases_are_not_a_collision(self) -> None: + """The same alias listed twice is one name, not two.""" + pg = get_dialect("postgres") + assert pg.alias_rewrite_map([LONG_EMAIL, LONG_EMAIL]) == pg.alias_rewrite_map([LONG_EMAIL]) + + def test_keys_and_values_are_disjoint(self) -> None: + """The invariant that makes the write pass safe: every key is OVER the + limit and every value is WITHIN it, so no substitution can produce + another key. (The two-phase pass defends this if it ever weakens.)""" + pg = get_dialect("postgres") + mapping = pg.alias_rewrite_map([LONG_NAME, LONG_EMAIL, "orders.status"]) + assert mapping + assert not (set(mapping) & set(mapping.values())) + for key, value in mapping.items(): + assert _nbytes(key) > 63 >= _nbytes(value) + + def test_digest_collision_raises(self, monkeypatch: pytest.MonkeyPatch) -> None: + """Force two distinct over-limit aliases onto one emitted name. + + Note the pair differs only in the MIDDLE — head and tail both survive + fitting, so the repro pair (which differs in its final segment) cannot + be used here: it stays distinct even with a constant digest, which is + exactly the readability property the shape is chosen for. + """ + import slayer.sql.dialects._identifier_fit as fitmod + + monkeypatch.setattr(fitmod, "_digest", lambda name: "deadbeef") + with pytest.raises(IdentifierCollisionError) as exc: + get_dialect("postgres").alias_rewrite_map([TWIN_A, TWIN_B]) + assert TWIN_A in str(exc.value) and TWIN_B in str(exc.value) + + def test_shortened_form_equal_to_an_existing_short_alias_raises(self) -> None: + """The guard must consider IDENTITY entries too. A short alias whose + spelling equals another alias's fitted form is a duplicate output name + that hash width alone cannot prevent.""" + pg = get_dialect("postgres") + collider = pg.fit_alias(LONG_EMAIL) # within 63 bytes, so an identity entry + assert pg.fit_alias(collider) == collider + with pytest.raises(IdentifierCollisionError): + pg.alias_rewrite_map([LONG_EMAIL, collider]) + + def test_error_names_the_dialect_and_limit(self, monkeypatch: pytest.MonkeyPatch) -> None: + import slayer.sql.dialects._identifier_fit as fitmod + + monkeypatch.setattr(fitmod, "_digest", lambda name: "deadbeef") + with pytest.raises(IdentifierCollisionError) as exc: + get_dialect("postgres").alias_rewrite_map([TWIN_A, TWIN_B]) + assert "postgres" in str(exc.value) and "63" in str(exc.value) + + def test_is_a_slayer_error_and_value_error(self) -> None: + from slayer.core.errors import SlayerError + + assert issubclass(IdentifierCollisionError, SlayerError) + assert issubclass(IdentifierCollisionError, ValueError) + + +# --------------------------------------------------------------------------- +# decode_result_keys — many-to-one must not silently overwrite +# --------------------------------------------------------------------------- + + +class TestDecodeCollision: + def test_two_keys_decoding_to_one_canonical_raises(self) -> None: + """A row carrying both the fitted form of an alias AND that alias's own + canonical spelling would silently lose one value on ``dict`` rebuild.""" + pg = get_dialect("postgres") + rows = [{pg.fit_alias(LONG_EMAIL): 1, LONG_EMAIL: 2}] + with pytest.raises(IdentifierCollisionError): + pg.decode_result_keys(rows, aliases=[LONG_EMAIL]) diff --git a/tests/integration/test_dev1756_identifier_length_pg.py b/tests/integration/test_dev1756_identifier_length_pg.py new file mode 100644 index 00000000..f325b800 --- /dev/null +++ b/tests/integration/test_dev1756_identifier_length_pg.py @@ -0,0 +1,307 @@ +"""DEV-1756 on a REAL Postgres server. + +The whole point of this issue is that byte-level emission tests pass while the +server rejects (or worse, silently mis-answers) the query. Postgres caps +identifiers at 63 bytes and truncates past it with only a NOTICE, so these +failures are invisible to any test that only inspects generated SQL. + +Two distinct failure modes are covered: + +* **Collapse** — two sibling aliases share a 63-byte prefix. With the DEV-1444 + outer wrap in play the re-projection is ambiguous (``AmbiguousColumnError``); + without it the two columns collapse into one in the result row. +* **Silent loss** — a SINGLE over-limit alias with no sibling. Postgres accepts + the query and returns the row keyed by the truncated name, so the engine's + canonical-alias lookup misses and a column silently disappears. No error is + raised anywhere, which makes this the more dangerous of the two. +""" + +import re +import uuid + +import pytest + +pytest.importorskip("pytest_postgresql") + +import psycopg +from pytest_postgresql import factories + +from slayer.async_utils import run_sync +from slayer.core.enums import DataType +from slayer.core.models import Column, DatasourceConfig, ModelJoin, SlayerModel +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.storage.yaml_storage import YAMLStorage + +postgresql_proc = factories.postgresql_proc(port=None) + +DS = "testpg" + +LONG_NAME = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name" +LONG_EMAIL = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email" + + +def _create_db(postgresql_proc): + info = postgresql_proc + db_name = f"test_{uuid.uuid4().hex[:12]}" + admin = psycopg.connect(host=info.host, port=info.port, user=info.user, dbname="postgres") + admin.autocommit = True + with admin.cursor() as cur: + cur.execute(f'CREATE DATABASE "{db_name}"') + admin.close() + conn = psycopg.connect(host=info.host, port=info.port, user=info.user, dbname=db_name) + return conn, db_name + + +def _drop_db(postgresql_proc, db_name): + info = postgresql_proc + admin = psycopg.connect(host=info.host, port=info.port, user=info.user, dbname="postgres") + admin.autocommit = True + with admin.cursor() as cur: + cur.execute(f'DROP DATABASE IF EXISTS "{db_name}" WITH (FORCE)') + admin.close() + + +@pytest.fixture(scope="module") +def _chain_storage(postgresql_proc, tmp_path_factory): + """3-hop join chain with model names long enough that the projection + aliases cross Postgres' 63-byte limit.""" + conn, db_name = _create_db(postgresql_proc) + try: + cur = conn.cursor() + cur.execute( + "CREATE TABLE consumers (id INTEGER PRIMARY KEY, name TEXT, email TEXT, " + "lifetime_value NUMERIC(10,2))" + ) + cur.execute("CREATE TABLE customers (id INTEGER PRIMARY KEY, consumer_id INTEGER)") + cur.execute("CREATE TABLE subscriptions (id INTEGER PRIMARY KEY, customer_id INTEGER)") + cur.execute( + "CREATE TABLE invoices (id INTEGER PRIMARY KEY, subscription_id INTEGER, " + "status TEXT, total_amount NUMERIC(10,2))" + ) + cur.executemany( + "INSERT INTO consumers VALUES (%s, %s, %s, %s)", + [(1, "Ann", "ann@example.io", 10), (2, "Bob", "bob@example.io", 20)], + ) + cur.executemany("INSERT INTO customers VALUES (%s, %s)", [(1, 1), (2, 2)]) + cur.executemany("INSERT INTO subscriptions VALUES (%s, %s)", [(1, 1), (2, 2)]) + cur.executemany( + "INSERT INTO invoices VALUES (%s, %s, %s, %s)", + [(1, 1, "paid", 100), (2, 1, "paid", 50), (3, 2, "paid", 200)], + ) + conn.commit() + + storage = YAMLStorage(base_dir=str(tmp_path_factory.mktemp("dev1756_pg"))) + info = postgresql_proc + run_sync(storage.save_datasource(DatasourceConfig( + name=DS, type="postgres", host=info.host, port=info.port, + database=db_name, username=info.user, password="", + ))) + + run_sync(storage.save_model(SlayerModel( + name="SandboxConsumer", sql_table="consumers", data_source=DS, + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="name", sql="name", type=DataType.TEXT), + Column(name="email", sql="email", type=DataType.TEXT), + Column(name="lifetimeValue", sql="lifetime_value", type=DataType.DOUBLE), + ], + ))) + run_sync(storage.save_model(SlayerModel( + name="SandboxCustomer", sql_table="customers", data_source=DS, + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="consumer_id", sql="consumer_id", type=DataType.INT), + ], + joins=[ModelJoin(target_model="SandboxConsumer", join_pairs=[["consumer_id", "id"]])], + ))) + run_sync(storage.save_model(SlayerModel( + name="SandboxSubscription", sql_table="subscriptions", data_source=DS, + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", sql="customer_id", type=DataType.INT), + ], + joins=[ModelJoin(target_model="SandboxCustomer", join_pairs=[["customer_id", "id"]])], + ))) + run_sync(storage.save_model(SlayerModel( + name="SandboxInvoiceV2", sql_table="invoices", data_source=DS, + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="status", sql="status", type=DataType.TEXT), + Column(name="totalAmount", sql="total_amount", type=DataType.DOUBLE), + Column(name="subscription_id", sql="subscription_id", type=DataType.INT), + ], + joins=[ModelJoin( + target_model="SandboxSubscription", join_pairs=[["subscription_id", "id"]], + )], + ))) + yield storage + finally: + conn.close() + _drop_db(postgresql_proc, db_name) + + +@pytest.fixture +def chain_env(_chain_storage): + """Per-test engine — the async SQLAlchemy engine binds to the event loop.""" + return SlayerQueryEngine(storage=_chain_storage) + + +DEEP = "SandboxSubscription.SandboxCustomer.SandboxConsumer" + + +@pytest.mark.integration +class TestPostgresIdentifierLength: + async def test_server_truncates_at_63_bytes(self, chain_env) -> None: + """Pin the premise against the actual server, so the rest of this file + cannot pass for the wrong reason if NAMEDATALEN ever differs.""" + client = chain_env._get_client( + await chain_env._resolve_datasource( + model=await chain_env.storage.get_model("SandboxInvoiceV2", data_source=DS), + ), + ("probe", "probe"), + ) + rows = await client.execute(sql=f'SELECT 1 AS "{LONG_EMAIL}"') + assert list(rows[0])[0] != LONG_EMAIL, "server did not truncate; premise broken" + assert len(list(rows[0])[0].encode()) == 63 + + async def test_collapse_with_outer_wrap(self, chain_env) -> None: + """The exact reported failure: two 73/74-byte siblings + an ORDER BY + hoist that forces the DEV-1444 outer wrap. Raised AmbiguousColumnError + before the fix.""" + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ + ColumnRef(name=f"{DEEP}.name"), + ColumnRef(name=f"{DEEP}.email"), + ColumnRef(name="status"), + ], + measures=[{"formula": "totalAmount:sum"}, {"formula": "*:count"}], + order=[OrderItem(column="totalAmount:avg", direction="desc")], + limit=10, + ) + result = await chain_env.execute(query=query) + # Exact rows: a fitted alias that resolved to the WRONG column would + # still be "present with distinct values", so pin the actual pairing. + got = { + (row[LONG_NAME], row[LONG_EMAIL], row["SandboxInvoiceV2.status"]): + (float(row["SandboxInvoiceV2.totalAmount_sum"]), row["SandboxInvoiceV2._count"]) + for row in result.data + } + assert got == { + ("Ann", "ann@example.io", "paid"): (150.0, 2), + ("Bob", "bob@example.io", "paid"): (200.0, 1), + } + + async def test_collapse_without_outer_wrap(self, chain_env) -> None: + """No ORDER BY, so no outer wrap and no server-side error — the two + columns would silently collapse into one result key.""" + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ + ColumnRef(name=f"{DEEP}.name"), + ColumnRef(name=f"{DEEP}.email"), + ], + measures=[{"formula": "*:count"}], + ) + result = await chain_env.execute(query=query) + assert result.data + names = {row[LONG_NAME] for row in result.data} + emails = {row[LONG_EMAIL] for row in result.data} + assert names == {"Ann", "Bob"} + assert emails == {"ann@example.io", "bob@example.io"} + + async def test_silent_loss_single_over_limit_alias(self, chain_env) -> None: + """The dangerous mode: ONE over-limit alias, no sibling to collide + with. Postgres accepts the query and keys the row by the truncated + name, so the column silently vanishes from the response.""" + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name=f"{DEEP}.email")], + measures=[{"formula": "totalAmount:sum"}], + ) + result = await chain_env.execute(query=query) + assert result.data + for row in result.data: + assert LONG_EMAIL in row, ( + f"over-limit alias missing from the result row: {sorted(row)}" + ) + assert row[LONG_EMAIL] in {"ann@example.io", "bob@example.io"} + + async def test_values_are_correct_not_merely_present(self, chain_env) -> None: + """A fitted alias that pointed at the wrong column would still be + 'present'. Pin the actual aggregate.""" + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name=f"{DEEP}.email")], + measures=[{"formula": "totalAmount:sum"}], + ) + result = await chain_env.execute(query=query) + totals = {row[LONG_EMAIL]: float(row["SandboxInvoiceV2.totalAmount_sum"]) for row in result.data} + assert totals == {"ann@example.io": 150.0, "bob@example.io": 200.0} + + async def test_response_columns_are_canonical(self, chain_env) -> None: + """Consumers must never see the shortened form — that is what the + read-side decode exists for.""" + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name=f"{DEEP}.name"), ColumnRef(name=f"{DEEP}.email")], + measures=[{"formula": "*:count"}], + ) + result = await chain_env.execute(query=query) + assert set(result.data[0]) >= {LONG_NAME, LONG_EMAIL} + # ...while the SQL that actually ran carries the fitted form. + assert LONG_EMAIL not in result.sql + + async def test_cached_execution_round_trip(self, chain_env) -> None: + """The cache stores the DECODED response and re-keys on the fitted + SQL; a hit must return canonical keys too.""" + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name=f"{DEEP}.email")], + measures=[{"formula": "*:count"}], + ) + first = await chain_env.execute(query=query, cache=True) + second = await chain_env.execute(query=query, cache=True) + # Canonical on BOTH the fresh result and the cache hit — proving the + # decode runs before storage, not only on the way out. + assert LONG_EMAIL in first.data[0] + assert LONG_EMAIL in second.data[0] + assert first.data == second.data + + async def test_deep_cross_model_measure_executes(self, chain_env) -> None: + """Surface 3: the `_cm_` CTE name is `_cm_` + the dotted alias, which + also crosses 63 bytes on this chain.""" + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": f"{DEEP}.lifetimeValue:sum"}], + ) + result = await chain_env.execute(query=query) + # The construct under test must actually be present... + assert "_cm_" in result.sql, "no cross-model CTE generated; test is vacuous" + for name in re.findall(r"\b_cm_\w+", result.sql): + assert len(name.encode()) <= 63, f"{name!r} exceeds the Postgres limit" + # ...and the value must be right, not merely non-empty. + alias = f"SandboxInvoiceV2.{DEEP}.lifetimeValue_sum" + assert len(result.data) == 1 + assert float(result.data[0][alias]) == 30.0 + + async def test_nested_query_backed_model_executes(self, chain_env) -> None: + """Surface 4: the virtual-model short names are emitted as output + column aliases and referenced by the outer stage. Mixed-case shorts + additionally exercise the case-folding regression.""" + stage1 = SlayerQuery( + name="stage1", + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name=f"{DEEP}.email")], + measures=[{"formula": "totalAmount:sum"}], + ) + stage2 = SlayerQuery( + source_model="stage1", + dimensions=[ColumnRef(name="SandboxSubscription__SandboxCustomer__SandboxConsumer__email")], + measures=[{"formula": "totalAmount_sum:sum"}], + ) + result = await chain_env.execute(query=[stage1, stage2]) + assert result.data + assert len(result.data) == 2 diff --git a/tests/test_dev1756_identifier_length.py b/tests/test_dev1756_identifier_length.py new file mode 100644 index 00000000..c7b9b699 --- /dev/null +++ b/tests/test_dev1756_identifier_length.py @@ -0,0 +1,868 @@ +"""DEV-1756: SLayer must bound generated identifiers to the dialect's limit. + +Postgres caps identifiers at 63 BYTES and SILENTLY truncates past it. SLayer's +projection aliases (``..``) cross that on a 3-hop +join, so two siblings collapse onto one effective output name: + + SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name 73 B + SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email 74 B + -> both truncate to ...SandboxCustomer.SandboxCon 63 B + +With the DEV-1444 outer wrap in play that raises ``AmbiguousColumnError``; +without it, the two columns silently collapse in the result row. + +Three surfaces are fixed here: + +1. Projection aliases -- QUOTED; inner SELECT, outer wrap, ORDER BY. +3. CTE names -- UNQUOTED; ``_cte_name_from_alias``. +4. Virtual-model shorts -- ``_query_as_model``'s ``_alias_to_short``. + +Surface 2 (join-path TABLE aliases such as +``SandboxSubscription__SandboxCustomer__SandboxConsumer``) is DEFERRED to +DEV-1743 and is deliberately NOT asserted on here — see ``_inscope_identifiers``. + +The primitive itself is unit-tested in ``tests/dialects/test_identifier_fit.py``; +live execution against a real server is in the Postgres integration suite. +""" + +from __future__ import annotations + +import pytest +import sqlglot +from sqlglot import exp + +from slayer.core.enums import DataType +from slayer.core.errors import IdentifierCollisionError +from slayer.core.models import Column, DatasourceConfig, ModelJoin, ModelMeasure, SlayerModel +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery +from slayer.engine.enriched import all_projection_aliases, public_projection_aliases +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.sql.dialects import get_dialect +from slayer.sql.dialects._alias_mangle import encode_alias +from slayer.sql.generator import SQLGenerator +from slayer.storage.yaml_storage import YAMLStorage + +DS = "sandbox" +DEEP = "SandboxSubscription.SandboxCustomer.SandboxConsumer" + +LONG_NAME = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name" +LONG_EMAIL = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email" + +# Two over-limit names differing ONLY in the middle — head and tail survive +# fitting identically, which is what makes a forced digest collision possible. +TWIN_A = "SandboxAlpha." * 3 + "111" + ".SandboxOmega" * 3 +TWIN_B = "SandboxAlpha." * 3 + "222" + ".SandboxOmega" * 3 + + +# --------------------------------------------------------------------------- +# Pre-change golden SQL — captured from the generator BEFORE this feature +# existed. These pin the "no churn for the common case" guarantee far more +# strongly than an idempotence check could. +# --------------------------------------------------------------------------- + +GOLDEN_SHORT_QUERY = { + "postgres": ( + 'SELECT\n SandboxInvoiceV2.status AS "SandboxInvoiceV2.status",\n' + ' SUM(SandboxInvoiceV2.total_amount) AS "SandboxInvoiceV2.totalAmount_sum"\n' + "FROM invoices AS SandboxInvoiceV2\nGROUP BY\n SandboxInvoiceV2.status" + ), + "bigquery": ( + "SELECT\n SandboxInvoiceV2.status AS `SandboxInvoiceV2___status`,\n" + " SUM(SandboxInvoiceV2.total_amount) AS `SandboxInvoiceV2___totalAmount_sum`\n" + "FROM invoices AS SandboxInvoiceV2\nGROUP BY\n SandboxInvoiceV2.status" + ), + "tsql": ( + "SELECT\n SandboxInvoiceV2.status AS [SandboxInvoiceV2___status],\n" + " SUM(SandboxInvoiceV2.total_amount) AS [SandboxInvoiceV2___totalAmount_sum]\n" + "FROM invoices AS SandboxInvoiceV2\nGROUP BY\n SandboxInvoiceV2.status" + ), + "mysql": ( + "SELECT\n SandboxInvoiceV2.status AS `SandboxInvoiceV2.status`,\n" + " SUM(SandboxInvoiceV2.total_amount) AS `SandboxInvoiceV2.totalAmount_sum`\n" + "FROM invoices AS SandboxInvoiceV2\nGROUP BY\n SandboxInvoiceV2.status" + ), + "sqlite": ( + 'SELECT\n SandboxInvoiceV2.status AS "SandboxInvoiceV2.status",\n' + ' SUM(SandboxInvoiceV2.total_amount) AS "SandboxInvoiceV2.totalAmount_sum"\n' + "FROM invoices AS SandboxInvoiceV2\nGROUP BY\n SandboxInvoiceV2.status" + ), +} + +# The full repro (long aliases + outer wrap) on an UNBOUNDED dialect: nothing +# may change, byte for byte. +GOLDEN_SQLITE_REPRO_ORDER = ( + 'SELECT\n "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name",\n' + ' "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email",\n' + ' "SandboxInvoiceV2.status",\n "SandboxInvoiceV2.totalAmount_sum",\n' + ' "SandboxInvoiceV2._count"\nFROM (\nSELECT\n' + " SandboxSubscription__SandboxCustomer__SandboxConsumer.name AS " + '"SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name",\n' + " SandboxSubscription__SandboxCustomer__SandboxConsumer.email AS " + '"SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email",\n' + ' SandboxInvoiceV2.status AS "SandboxInvoiceV2.status",\n' + ' SUM(SandboxInvoiceV2.total_amount) AS "SandboxInvoiceV2.totalAmount_sum",\n' + ' COUNT(*) AS "SandboxInvoiceV2._count",\n' + ' AVG(SandboxInvoiceV2.total_amount) AS "SandboxInvoiceV2.totalAmount_avg"\n' + "FROM invoices AS SandboxInvoiceV2\nLEFT JOIN subscriptions AS SandboxSubscription\n" + " ON SandboxInvoiceV2.subscription_id = SandboxSubscription.id\n" + "LEFT JOIN customers AS SandboxSubscription__SandboxCustomer\n" + " ON SandboxSubscription.customer_id = SandboxSubscription__SandboxCustomer.id\n" + "LEFT JOIN consumers AS SandboxSubscription__SandboxCustomer__SandboxConsumer\n" + " ON SandboxSubscription__SandboxCustomer.consumer_id = " + "SandboxSubscription__SandboxCustomer__SandboxConsumer.id\nGROUP BY\n" + " SandboxSubscription__SandboxCustomer__SandboxConsumer.name,\n" + " SandboxSubscription__SandboxCustomer__SandboxConsumer.email,\n" + " SandboxInvoiceV2.status\n) AS _outer\nORDER BY\n" + ' "SandboxInvoiceV2.totalAmount_avg" DESC\nLIMIT 10' +) + + +# --------------------------------------------------------------------------- +# Fixtures — the reported 3-hop chain, with realistic model-name lengths +# --------------------------------------------------------------------------- + + +def _chain_models(*, case_colliding_columns: bool = False) -> list[SlayerModel]: + consumer_columns = [ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="name", sql="name", type=DataType.TEXT), + Column(name="email", sql="email", type=DataType.TEXT), + Column(name="lifetimeValue", sql="lifetime_value", type=DataType.DOUBLE), + ] + if case_colliding_columns: + # Differs from ``email`` only by case: the derived virtual-model shorts + # are emitted into a namespace Postgres case-folds. + consumer_columns.append(Column(name="Email", sql="email", type=DataType.TEXT)) + return [ + SlayerModel( + name="SandboxInvoiceV2", sql_table="invoices", data_source=DS, + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="status", sql="status", type=DataType.TEXT), + Column(name="totalAmount", sql="total_amount", type=DataType.DOUBLE), + Column(name="subscription_id", sql="subscription_id", type=DataType.INT), + ], + joins=[ModelJoin( + target_model="SandboxSubscription", join_pairs=[["subscription_id", "id"]], + )], + ), + SlayerModel( + name="SandboxSubscription", sql_table="subscriptions", data_source=DS, + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", sql="customer_id", type=DataType.INT), + ], + joins=[ModelJoin(target_model="SandboxCustomer", join_pairs=[["customer_id", "id"]])], + ), + SlayerModel( + name="SandboxCustomer", sql_table="customers", data_source=DS, + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="consumer_id", sql="consumer_id", type=DataType.INT), + ], + joins=[ModelJoin(target_model="SandboxConsumer", join_pairs=[["consumer_id", "id"]])], + ), + SlayerModel( + name="SandboxConsumer", sql_table="consumers", data_source=DS, + columns=consumer_columns, + ), + ] + + +async def _build_engine(tmp_path, **kw) -> tuple[SlayerQueryEngine, SlayerModel]: + storage = YAMLStorage(base_dir=str(tmp_path)) + await storage.save_datasource(DatasourceConfig( + name=DS, type="postgres", host="localhost", port=5432, + database="x", username="u", password="p", + )) + models = _chain_models(**kw) + for m in models: + await storage.save_model(m) + return SlayerQueryEngine(storage=storage), models[0] + + +@pytest.fixture +async def chain(tmp_path): + """Engine + root model for the 3-hop chain, with a postgres datasource.""" + return await _build_engine(tmp_path) + + +def _repro_query(*, with_order: bool = False) -> SlayerQuery: + """The exact query from the DEV-1756 report.""" + kwargs = {} + if with_order: + kwargs["order"] = [OrderItem(column="totalAmount:avg", direction="desc")] + kwargs["limit"] = 10 + return SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ + ColumnRef(name=f"{DEEP}.name"), + ColumnRef(name=f"{DEEP}.email"), + ColumnRef(name="status"), + ], + measures=[{"formula": "totalAmount:sum"}, {"formula": "*:count"}], + **kwargs, + ) + + +SHORT_QUERY = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "totalAmount:sum"}], +) + + +# --------------------------------------------------------------------------- +# Identifier-inspection helpers +# +# Scope note: these deliberately inspect ONLY the namespaces this issue fixes — +# output-column aliases, ORDER BY references and CTE names. Join-path TABLE +# aliases (``exp.TableAlias``) are surface 2, deferred to DEV-1743, and are +# excluded so a future long join chain fails THERE rather than confusingly here. +# --------------------------------------------------------------------------- + + +def _nbytes(s: str) -> int: + return len(s.encode("utf-8")) + + +def _pg_effective(name: str, *, quoted: bool) -> str: + """What Postgres actually resolves an identifier to: truncate to 63 bytes, + and additionally case-fold when it was written unquoted.""" + clipped = name.encode("utf-8")[:63].decode("utf-8", "ignore") + return clipped if quoted else clipped.lower() + + +def _projection_aliases(select: exp.Select) -> list[tuple[str, bool]]: + """(name, quoted) for each output column of one SELECT scope.""" + names: list[tuple[str, bool]] = [] + for proj in select.expressions: + if isinstance(proj, exp.Alias) and isinstance(proj.args.get("alias"), exp.Identifier): + ident = proj.args["alias"] + names.append((ident.this, bool(ident.quoted))) + elif isinstance(proj, exp.Column) and isinstance(proj.this, exp.Identifier): + names.append((proj.this.this, bool(proj.this.quoted))) + return names + + +def _order_by_refs(select: exp.Select) -> list[tuple[str, bool]]: + """(name, quoted) for each column referenced in one SELECT's ORDER BY.""" + order = select.args.get("order") + if order is None: + return [] + return [ + (col.this.this, bool(col.this.quoted)) + for col in order.find_all(exp.Column) + if isinstance(col.this, exp.Identifier) + ] + + +def _cte_names(tree: exp.Expression) -> list[tuple[str, bool]]: + return [ + (c.alias_or_name, bool(getattr(c.args.get("alias"), "quoted", False))) + for c in tree.find_all(exp.CTE) + ] + + +def _cte_table_refs(tree: exp.Expression) -> set[str]: + """Names of every table reference, so a CTE reference can be matched + against its definition EXACTLY rather than by substring count.""" + return { + t.this.this + for t in tree.find_all(exp.Table) + if isinstance(t.this, exp.Identifier) + } + + +def _inscope_identifiers(sql: str, dialect: str = "postgres") -> list[tuple[str, bool]]: + """Every identifier in a namespace this issue owns (see scope note).""" + tree = sqlglot.parse_one(sql, dialect=dialect) + out: list[tuple[str, bool]] = [] + for select in tree.find_all(exp.Select): + out.extend(_projection_aliases(select)) + out.extend(_order_by_refs(select)) + out.extend(_cte_names(tree)) + return out + + +def _assert_within_limit(sql: str, limit: int, dialect: str = "postgres") -> None: + for name, _ in _inscope_identifiers(sql, dialect): + assert _nbytes(name) <= limit, f"{name!r} is {_nbytes(name)} bytes\n{sql}" + + +def _assert_no_namespace_collision(sql: str, dialect: str = "postgres") -> None: + """Within each namespace, identifiers must stay distinct AFTER the backend's + normalization (truncate, plus case-fold when unquoted) — not merely be + short enough.""" + tree = sqlglot.parse_one(sql, dialect=dialect) + namespaces: list[tuple[str, list[tuple[str, bool]]]] = [ + (f"select#{i}", _projection_aliases(select)) + for i, select in enumerate(tree.find_all(exp.Select)) + ] + namespaces.append(("cte", _cte_names(tree))) + for ns, names in namespaces: + seen: dict[str, str] = {} + for name, quoted in names: + eff = _pg_effective(name, quoted=quoted) + prior = seen.get(eff) + assert prior is None or prior == name, ( + f"{ns}: {prior!r} and {name!r} both normalize to {eff!r}\n{sql}" + ) + seen[eff] = name + + +def _assert_order_by_refs_resolve(sql: str, dialect: str = "postgres") -> None: + """Every quoted ORDER BY reference must name a projection alias that + actually exists somewhere in the statement. This is the pairing check that + catches an ORDER BY left pointing at an unfitted alias.""" + tree = sqlglot.parse_one(sql, dialect=dialect) + projected = {n for select in tree.find_all(exp.Select) for n, _ in _projection_aliases(select)} + for select in tree.find_all(exp.Select): + for name, quoted in _order_by_refs(select): + if quoted: + assert name in projected, ( + f"ORDER BY references {name!r}, which no SELECT projects\n{sql}" + ) + + +async def _sql(engine, model, query, *, dialect: str = "postgres", mode: str = "outer") -> str: + enriched = await engine._enrich(query=query, model=model) + return SQLGenerator(dialect=dialect).generate(enriched=enriched, render_mode=mode) + + +# =========================================================================== +# 1. The premise — without the fix these aliases really do collide +# =========================================================================== + + +class TestPremise: + async def test_canonical_aliases_are_over_the_limit(self, chain) -> None: + engine, model = chain + enriched = await engine._enrich(query=_repro_query(), model=model) + aliases = public_projection_aliases(enriched) + assert LONG_NAME in aliases and LONG_EMAIL in aliases + assert _nbytes(LONG_NAME) == 73 and _nbytes(LONG_EMAIL) == 74 + + def test_the_two_aliases_share_a_63_byte_prefix(self) -> None: + assert LONG_NAME.encode()[:63] == LONG_EMAIL.encode()[:63] + + +# =========================================================================== +# 2. Surface 1 — projection aliases +# =========================================================================== + + +class TestProjectionAliases: + async def test_repro_emits_no_over_limit_identifier(self, chain) -> None: + engine, model = chain + _assert_within_limit(await _sql(engine, model, _repro_query()), 63) + + async def test_repro_has_no_namespace_collision(self, chain) -> None: + engine, model = chain + _assert_no_namespace_collision(await _sql(engine, model, _repro_query())) + + async def test_repro_still_parses(self, chain) -> None: + engine, model = chain + sql = await _sql(engine, model, _repro_query()) + assert len(sqlglot.parse(sql, dialect="postgres")) == 1 + + async def test_outer_wrap_uses_the_same_token_everywhere(self, chain) -> None: + """The reported failure: the inner ``AS ``, the outer wrap's + projection and the ORDER BY must all carry the IDENTICAL identifier.""" + engine, model = chain + sql = await _sql(engine, model, _repro_query(with_order=True)) + assert ") AS _outer" in sql, "outer wrap did not fire; test is vacuous" + tree = sqlglot.parse_one(sql, dialect="postgres") + selects = list(tree.find_all(exp.Select)) + outer_names = {n for n, _ in _projection_aliases(selects[0])} + inner_names = {n for n, _ in _projection_aliases(selects[1])} + fitted = get_dialect("postgres").fit_alias(LONG_EMAIL) + assert fitted in outer_names and fitted in inner_names + _assert_within_limit(sql, 63) + _assert_no_namespace_collision(sql) + _assert_order_by_refs_resolve(sql) + + async def test_canonical_alias_does_not_survive_anywhere(self, chain) -> None: + """Pairing check: if ANY occurrence were missed, the definition and its + references would disagree. Stronger than a max-length assertion.""" + engine, model = chain + sql = await _sql(engine, model, _repro_query(with_order=True)) + assert LONG_NAME not in sql + assert LONG_EMAIL not in sql + + @pytest.mark.parametrize("dialect", sorted(GOLDEN_SHORT_QUERY)) + async def test_under_limit_query_is_byte_identical(self, chain, dialect: str) -> None: + """No churn for the 99% case, pinned against SQL captured BEFORE this + feature existed.""" + engine, model = chain + assert await _sql(engine, model, SHORT_QUERY, dialect=dialect) == GOLDEN_SHORT_QUERY[dialect] + + async def test_unbounded_dialect_output_is_byte_identical(self, chain) -> None: + """SQLite has no limit, so even the full repro — long aliases, outer + wrap, ORDER BY — must be untouched byte for byte.""" + engine, model = chain + sql = await _sql(engine, model, _repro_query(with_order=True), dialect="sqlite") + assert sql == GOLDEN_SQLITE_REPRO_ORDER + + @pytest.mark.parametrize("dialect", ["clickhouse", "trino", "databricks"]) + async def test_other_unbounded_dialects_keep_the_long_alias(self, chain, dialect: str) -> None: + engine, model = chain + sql = await _sql(engine, model, _repro_query(with_order=True), dialect=dialect) + # Inner AS + outer projection: both untouched. + assert sql.count(LONG_EMAIL) == 2 + + async def test_wrapped_render_mode_also_fitted(self, chain) -> None: + """``render_mode='wrapped'`` feeds ``_query_as_model``; its inner + aliases are just as subject to truncation.""" + engine, model = chain + _assert_within_limit(await _sql(engine, model, _repro_query(), mode="wrapped"), 63) + + +class TestManglingDialects: + """BigQuery / T-SQL already mangle dotted aliases; length-fitting must + compose with that, not fight it.""" + + @pytest.mark.parametrize("dialect,limit", [("bigquery", 300), ("tsql", 128)]) + async def test_long_alias_is_mangled_and_fitted(self, chain, dialect: str, limit: int) -> None: + engine, model = chain + sql = await _sql(engine, model, _repro_query(), dialect=dialect) + _assert_within_limit(sql, limit, dialect=dialect) + assert LONG_EMAIL not in sql + + async def test_emit_alias_matches_what_is_in_the_sql(self, chain) -> None: + """``emit_alias`` is what the decode map is built from, so it must be + exactly the token the SQL carries.""" + engine, model = chain + for dialect in ("postgres", "bigquery", "tsql", "mysql"): + sql = await _sql(engine, model, _repro_query(), dialect=dialect) + names = {n for n, _ in _inscope_identifiers(sql, dialect)} + assert get_dialect(dialect).emit_alias(LONG_EMAIL) in names, dialect + + async def test_mangling_is_not_applied_twice(self, chain) -> None: + """The base length pass runs BEFORE the dot-mangle regex. If it emitted + an already-mangled form the regex would double-encode ``___`` to + ``______``.""" + engine, model = chain + sql = await _sql(engine, model, _repro_query(), dialect="bigquery") + fitted = get_dialect("bigquery").fit_alias(LONG_EMAIL) + assert encode_alias(fitted) in sql + assert encode_alias(encode_alias(fitted)) not in sql + + +# =========================================================================== +# 3. Read side +# =========================================================================== + + +class TestDecodeResultKeys: + def test_shortened_keys_restored_to_canonical(self) -> None: + pg = get_dialect("postgres") + rows = [{pg.emit_alias(LONG_EMAIL): "a@b.io", "SandboxInvoiceV2.status": "paid"}] + got = pg.decode_result_keys(rows, aliases=[LONG_EMAIL, "SandboxInvoiceV2.status"]) + assert got == [{LONG_EMAIL: "a@b.io", "SandboxInvoiceV2.status": "paid"}] + + def test_identity_when_nothing_shortened(self) -> None: + rows = [{"orders.status": "paid"}] + assert get_dialect("postgres").decode_result_keys(rows, aliases=["orders.status"]) == rows + + def test_identity_without_aliases(self) -> None: + rows = [{"whatever": 1}] + assert get_dialect("postgres").decode_result_keys(rows) == rows + + def test_unknown_key_passes_through(self) -> None: + rows = [{"surprise": 1}] + assert get_dialect("postgres").decode_result_keys(rows, aliases=[LONG_EMAIL]) == rows + + def test_empty_rows(self) -> None: + assert get_dialect("postgres").decode_result_keys([], aliases=[LONG_EMAIL]) == [] + + def test_hidden_alias_is_decoded_too(self) -> None: + """Hidden ORDER-BY hoists are projected in the inner SELECT, so a row + can legitimately carry one; it must decode like any other.""" + pg = get_dialect("postgres") + hidden = LONG_EMAIL.replace(".email", ".totalAmount_avg") + rows = [{pg.emit_alias(hidden): 1.0}] + assert pg.decode_result_keys(rows, aliases=[hidden]) == [{hidden: 1.0}] + + def test_bigquery_reverses_both_manglings(self) -> None: + bq = get_dialect("bigquery") + long_dotted = ".".join(["Sandbox" * 6] * 8) + rows = [{bq.emit_alias(long_dotted): 1}] + assert bq.decode_result_keys(rows, aliases=[long_dotted]) == [{long_dotted: 1}] + + def test_bigquery_falls_back_to_dot_decode_outside_the_map(self) -> None: + """Keys not in the length map must still get today's ``___`` -> ``.`` + treatment, or short-alias BigQuery results would regress.""" + bq = get_dialect("bigquery") + assert bq.decode_result_keys([{"orders___status": 1}], aliases=[]) == [{"orders.status": 1}] + + +class TestDecodeWiring: + """The decode must be handed the FULL alias set, hidden entries included — + an implementation that used only the public aliases would still pass the + end-to-end repro.""" + + async def test_run_and_build_passes_all_projection_aliases( + self, chain, monkeypatch: pytest.MonkeyPatch, + ) -> None: + from slayer.sql.dialects.postgres import PostgresDialect + + engine, _ = chain + prepared = await engine._prepare_pipeline( + query=_repro_query(with_order=True), named_queries={}, runtime_kwarg={}, + ) + expected = all_projection_aliases(prepared.enriched) + hidden = [a for a in expected if a not in public_projection_aliases(prepared.enriched)] + assert hidden, "fixture must produce a hidden alias or this is vacuous" + + seen: dict[str, object] = {} + + def _spy(self, rows, *, aliases=()): + seen["aliases"] = list(aliases) + return rows + + monkeypatch.setattr(PostgresDialect, "decode_result_keys", _spy) + + class _FakeClient: + async def execute(self, sql): + return [] + + await engine._run_and_build(prepared=prepared, client=_FakeClient()) + assert seen["aliases"] == expected + + +# =========================================================================== +# 4. The alias set +# =========================================================================== + + +class TestAllProjectionAliases: + async def test_includes_public_aliases(self, chain) -> None: + engine, model = chain + enriched = await engine._enrich(query=_repro_query(), model=model) + every = all_projection_aliases(enriched) + for a in public_projection_aliases(enriched): + assert a in every + + async def test_includes_hidden_order_by_hoist(self, chain) -> None: + """``totalAmount:avg`` is projected in the inner SELECT purely to + satisfy ORDER BY. It truncates like any other alias, so the pass must + see it.""" + engine, model = chain + enriched = await engine._enrich(query=_repro_query(with_order=True), model=model) + public = public_projection_aliases(enriched) + hidden = [a for a in all_projection_aliases(enriched) if a not in public] + assert any("totalAmount_avg" in a for a in hidden), hidden + + async def test_is_deterministic(self, chain) -> None: + engine, model = chain + enriched = await engine._enrich(query=_repro_query(), model=model) + assert all_projection_aliases(enriched) == all_projection_aliases(enriched) + + +# =========================================================================== +# 5. Surface 3 — CTE names +# =========================================================================== + + +def _deep_cross_model_query(*, two_measures: bool = False) -> SlayerQuery: + measures = [ModelMeasure(formula=f"{DEEP}.lifetimeValue:sum")] + if two_measures: + measures.append(ModelMeasure(formula=f"{DEEP}.lifetimeValue:avg")) + return SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name="status")], + measures=measures, + ) + + +class TestCteNames: + async def test_cross_model_cte_name_within_limit(self, chain) -> None: + engine, model = chain + sql = await _sql(engine, model, _deep_cross_model_query()) + assert "_cm_" in sql, "no cross-model CTE was generated; test is vacuous" + _assert_within_limit(sql, 63) + + async def test_cte_definition_and_references_agree(self, chain) -> None: + """A CTE name is emitted UNQUOTED, so a truncated definition and an + untruncated reference would silently disagree. Compare parsed + identifiers, not substring counts.""" + engine, model = chain + sql = await _sql(engine, model, _deep_cross_model_query()) + tree = sqlglot.parse_one(sql, dialect="postgres") + defined = [n for n, _ in _cte_names(tree)] + assert defined, "no CTE generated; test is vacuous" + referenced = _cte_table_refs(tree) + for name in defined: + assert _nbytes(name) <= 63 + assert name in referenced, f"CTE {name!r} defined but never referenced\n{sql}" + + async def test_two_deep_cross_model_ctes_stay_distinct(self, chain) -> None: + """Two deep cross-model measures produce two over-limit CTE names; both + must fit AND stay distinct after the server's truncation.""" + engine, model = chain + sql = await _sql(engine, model, _deep_cross_model_query(two_measures=True)) + tree = sqlglot.parse_one(sql, dialect="postgres") + names = [n for n, _ in _cte_names(tree)] + assert len(names) >= 2, f"expected two cross-model CTEs\n{sql}" + effective = [_pg_effective(n, quoted=False) for n in names] + assert len(set(effective)) == len(effective), effective + _assert_within_limit(sql, 63) + + def test_cte_namespace_collision_raises(self, monkeypatch: pytest.MonkeyPatch) -> None: + """Forced digest collision in the CTE namespace must raise rather than + emit two identically-named CTEs — which would silently make one of them + reference the other's rows. + + Driven through the allocator directly: CTE names derive from measure + aliases, and two aliases reachable from one query always differ in + their final segment, which fitting preserves. So a colliding PAIR is + not constructible from a natural query — the allocator is the thing + that has to hold the invariant. + """ + import slayer.sql.dialects._identifier_fit as fitmod + + gen = SQLGenerator(dialect="postgres") + monkeypatch.setattr(fitmod, "_digest", lambda name: "deadbeef") + first = gen._cte_name("_cm_", TWIN_A) + assert _nbytes(first) <= 63 + with pytest.raises(IdentifierCollisionError) as exc: + gen._cte_name("_cm_", TWIN_B) + assert "CTE name" in str(exc.value) + + def test_cte_allocator_is_idempotent_for_the_same_owner(self) -> None: + """Several code paths re-derive a CTE's name in order to reference it; + that must not read as a collision.""" + gen = SQLGenerator(dialect="postgres") + assert gen._cte_name("_cm_", TWIN_A) == gen._cte_name("_cm_", TWIN_A) + + def test_cte_allocator_resets_per_statement(self) -> None: + """One generator instance generates many statements; allocation is + per-statement, so names must not accumulate across calls.""" + gen = SQLGenerator(dialect="postgres") + gen._cte_name("_cm_", TWIN_A) + gen._cte_names = {} + gen._cte_name("_cm_", TWIN_A) # would raise if state leaked wrongly + + def test_cte_name_helper_is_pure_and_bounded(self) -> None: + from slayer.sql.generator import _cte_name_from_alias + + long_alias = LONG_EMAIL + ".lifetimeValue_sum" + a = _cte_name_from_alias("_cm_", long_alias, limit=63) + b = _cte_name_from_alias("_cm_", long_alias, limit=63) + assert a == b + assert _nbytes(a) <= 63 + + def test_cte_name_helper_counts_the_prefix(self) -> None: + """The budget covers the WHOLE emitted name, prefix included — a + ``cp_value_12_`` prefix eats 12 of the 63 bytes.""" + from slayer.sql.generator import _cte_name_from_alias + + assert _nbytes(_cte_name_from_alias("cp_value_12_", "a" * 60, limit=63)) <= 63 + + def test_cte_name_helper_unbounded(self) -> None: + """``limit=None`` keeps today's behaviour exactly. The alias here is + already flat, so sanitization is a no-op and the result is a plain + concatenation.""" + from slayer.sql.generator import _cte_name_from_alias + + assert _cte_name_from_alias("_cm_", "a" * 200, limit=None) == "_cm_" + "a" * 200 + + def test_cte_name_helper_still_sanitizes(self) -> None: + from slayer.sql.generator import _cte_name_from_alias + + assert _cte_name_from_alias("_cm_", "a.b", limit=None) == "_cm_a__b" + + def test_distinct_long_aliases_get_distinct_cte_names(self) -> None: + from slayer.sql.generator import _cte_name_from_alias + + a = _cte_name_from_alias("_cm_", LONG_NAME + ".v_sum", limit=63) + b = _cte_name_from_alias("_cm_", LONG_EMAIL + ".v_sum", limit=63) + assert a != b + + def test_cte_name_is_a_legal_unquoted_identifier(self) -> None: + import re + + from slayer.sql.generator import _cte_name_from_alias + + got = _cte_name_from_alias("_cm_", LONG_EMAIL + ".lifetimeValue_sum", limit=63) + assert re.match(r"^[A-Za-z_][A-Za-z0-9_]*$", got), got + + +# =========================================================================== +# 6. Surface 4 — _query_as_model short names +# =========================================================================== + + +def _wrapper_select(vm_sql: str) -> exp.Select: + """The outermost SELECT of a virtual model's SQL — the rename wrapper.""" + return next(iter(sqlglot.parse_one(vm_sql, dialect="postgres").find_all(exp.Select))) + + +class TestVirtualModelShorts: + async def test_short_within_limit_and_matches_column_name(self, chain) -> None: + engine, _ = chain + vm = await engine._query_as_model(inner_query=_repro_query()) + emitted = {n for n, _ in _projection_aliases(_wrapper_select(vm.sql))} + for col in vm.columns: + assert _nbytes(col.name) <= 63, col.name + assert col.name in emitted, ( + f"virtual-model column {col.name!r} is not an alias the wrapper " + f"emits\n{vm.sql}" + ) + + async def test_long_siblings_get_distinct_shorts(self, chain) -> None: + engine, _ = chain + vm = await engine._query_as_model(inner_query=_repro_query()) + names = [c.name for c in vm.columns] + assert len(names) == len(set(names)) + + async def test_short_aliases_are_quoted(self, chain) -> None: + """Emitted bare, a mixed-case short is case-folded by Postgres while + the outer stage references it quoted -> UndefinedColumnError. Quoting + the alias fixes that AND removes the case-fold collision exposure.""" + engine, _ = chain + vm = await engine._query_as_model(inner_query=_repro_query()) + select = _wrapper_select(vm.sql) + assert select.expressions, vm.sql + for proj in select.expressions: + assert isinstance(proj, exp.Alias), f"{proj.sql()} is not an aliased projection" + ident = proj.args.get("alias") + assert isinstance(ident, exp.Identifier) and ident.quoted, ( + f"short alias {ident} must be dialect-quoted\n{vm.sql}" + ) + + async def test_inner_and_wrapper_agree_on_the_fitted_alias(self, chain) -> None: + """The inner SQL is fitted by ``generate()``; the wrapper references + those aliases. They must not drift.""" + engine, _ = chain + vm = await engine._query_as_model(inner_query=_repro_query()) + fitted = get_dialect("postgres").fit_alias(LONG_EMAIL) + select = _wrapper_select(vm.sql) + sources = { + proj.this.this.this + for proj in select.expressions + if isinstance(proj, exp.Alias) + and isinstance(proj.this, exp.Column) + and isinstance(proj.this.this, exp.Identifier) + } + assert fitted in sources, f"wrapper does not reference the fitted alias\n{vm.sql}" + assert LONG_EMAIL not in vm.sql + + async def test_case_colliding_shorts_raise(self, tmp_path) -> None: + """``email`` and ``Email`` on the deepest model produce shorts that + differ only by case. They are emitted into a namespace Postgres + case-folds, so this must be caught, not silently collapsed.""" + engine, _ = await _build_engine(tmp_path, case_colliding_columns=True) + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ + ColumnRef(name=f"{DEEP}.email"), + ColumnRef(name=f"{DEEP}.Email"), + ], + measures=[{"formula": "*:count"}], + ) + with pytest.raises(IdentifierCollisionError): + await engine._query_as_model(inner_query=query) + + async def test_nested_dag_two_levels_agree(self, chain) -> None: + """Two stages: stage 2 references stage 1's virtual-model columns.""" + engine, _ = chain + stage1 = _repro_query().model_copy(update={"name": "stage1"}) + vm = await engine._query_as_model(inner_query=stage1, override_name="stage1") + short = next(c.name for c in vm.columns if "email" in c.name.lower()) + stage2 = SlayerQuery( + source_model="stage1", + dimensions=[ColumnRef(name=short)], + measures=[{"formula": "totalAmount_sum:sum"}], + ) + resp = await engine.execute(query=[stage1, stage2], dry_run=True) + _assert_within_limit(resp.sql, 63) + _assert_no_namespace_collision(resp.sql) + _assert_order_by_refs_resolve(resp.sql) + + +# =========================================================================== +# 7. Engine-level contract — consumers never see the shortened form +# =========================================================================== + + +class TestEngineContract: + async def test_dry_run_columns_stay_canonical(self, chain) -> None: + engine, _ = chain + resp = await engine.execute(query=_repro_query(), dry_run=True) + assert LONG_NAME in resp.columns + assert LONG_EMAIL in resp.columns + + async def test_dry_run_sql_carries_the_shortened_form(self, chain) -> None: + engine, _ = chain + resp = await engine.execute(query=_repro_query(), dry_run=True) + assert LONG_EMAIL not in resp.sql + assert get_dialect("postgres").fit_alias(LONG_EMAIL) in resp.sql + + async def test_attribute_keys_are_result_keys(self, chain) -> None: + engine, _ = chain + resp = await engine.execute(query=_repro_query(), dry_run=True) + for key in resp.attributes.dimensions: + assert key in resp.columns + + +# =========================================================================== +# 8. Sweep — every generator shape, not just the reported one +# =========================================================================== + + +class TestSweep: + """Aliases can be synthesized in the generator rather than stored on the + enriched buckets. Assert PAIRING (no canonical over-limit alias string + survives) across every SQL shape the generator can build.""" + + @pytest.fixture + def queries(self) -> list[SlayerQuery]: + return [ + _repro_query(), # plain dims + measures + _repro_query(with_order=True), # ORDER BY hoist + outer wrap + _deep_cross_model_query(), # cross-model CTE (_cm_ + WITH) + SlayerQuery( # arithmetic expression + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name=f"{DEEP}.email")], + measures=[ + {"formula": "totalAmount:sum"}, + {"formula": "totalAmount:sum / *:count", "name": "avg_ticket"}, + ], + ), + SlayerQuery( # filter on a long dotted dim + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name=f"{DEEP}.name")], + measures=[{"formula": "*:count"}], + filters=[f"{DEEP}.email IS NOT NULL"], + ), + ] + + async def test_no_over_limit_identifier_anywhere(self, chain, queries) -> None: + engine, model = chain + for q in queries: + _assert_within_limit(await _sql(engine, model, q), 63) + + async def test_no_namespace_collision_anywhere(self, chain, queries) -> None: + engine, model = chain + for q in queries: + _assert_no_namespace_collision(await _sql(engine, model, q)) + + async def test_order_by_refs_resolve_anywhere(self, chain, queries) -> None: + engine, model = chain + for q in queries: + _assert_order_by_refs_resolve(await _sql(engine, model, q)) + + async def test_no_canonical_over_limit_alias_survives(self, chain, queries) -> None: + engine, model = chain + for q in queries: + enriched = await engine._enrich(query=q, model=model) + sql = SQLGenerator(dialect="postgres").generate(enriched=enriched) + over = [a for a in all_projection_aliases(enriched) if _nbytes(a) > 63] + assert over, "sweep entry has no over-limit alias; it proves nothing" + for alias in over: + assert alias not in sql, ( + f"canonical alias {alias!r} still present — some emission " + f"site was not rewritten\n{sql}" + ) diff --git a/tests/test_query_backed_models.py b/tests/test_query_backed_models.py index 105fcb57..0d988b43 100644 --- a/tests/test_query_backed_models.py +++ b/tests/test_query_backed_models.py @@ -1418,15 +1418,17 @@ async def test_inner_stage_aggregated_measure_honors_user_name(self) -> None: f"expected 'rev_sum' in cached columns, got: {col_names}" ) sql = loaded.backing_query_sql or "" - # Inner-stage wrap renames `"orders.rev" AS rev`; loose match on - # the alias keyword + name (newline-tolerant). + # Inner-stage wrap renames `"orders.rev" AS "rev"`; loose match on + # the alias keyword + name (newline-tolerant). DEV-1756: the short + # is now ALWAYS dialect-quoted, so an unquoted mixed-case short is + # no longer case-folded out from under the outer stage's reference. import re - assert re.search(r"\bAS\s+rev\b", sql), ( - f"expected inner-stage 'AS rev' rename in SQL:\n{sql}" + assert re.search(r'\bAS\s+"rev"', sql), ( + f"expected inner-stage 'AS \"rev\"' rename in SQL:\n{sql}" ) # The canonical name must not leak into the wrapped subquery's # exposed alias. - assert not re.search(r"\bAS\s+amount_sum\b", sql), ( + assert not re.search(r'\bAS\s+"?amount_sum"?', sql), ( f"canonical 'amount_sum' must not be the surfaced inner alias:\n{sql}" ) finally: @@ -1491,7 +1493,8 @@ async def test_query_as_model_emits_user_alias_unit(self) -> None: f"'name', got: {col_names}" ) import re - assert re.search(r"\bAS\s+rev\b", virtual.sql), ( + # DEV-1756: shorts are always dialect-quoted now. + assert re.search(r'\bAS\s+"rev"', virtual.sql), ( f"wrapped SQL must rename to user alias 'rev':\n{virtual.sql}" ) finally: diff --git a/tests/test_sql_generator.py b/tests/test_sql_generator.py index fa83d15a..83467867 100644 --- a/tests/test_sql_generator.py +++ b/tests/test_sql_generator.py @@ -7011,9 +7011,9 @@ def trim_spy(self, *, sql, enriched): call_order.append("trim") return real_trim(self, sql=sql, enriched=enriched) - def rewrite_spy(self, sql): + def rewrite_spy(self, sql, **kwargs): call_order.append("rewrite") - return real_rewrite(self, sql) + return real_rewrite(self, sql, **kwargs) with patch.object(SQLGenerator, "_apply_outer_projection_trim", trim_spy), \ patch.object(BigqueryDialect, "rewrite_emitted_sql", rewrite_spy): From d72c70d4bdd0160e6ffba197dd2e4e6ba8ea9db1 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 15:54:26 +0200 Subject: [PATCH 02/10] fix(DEV-1756): close three holes in the identifier-collision checks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings from Codex on PR #289, all in the same family: a collision check that could not observe the collision it was meant to catch. 1. `_query_as_model` keyed its short-name allocation by the short itself (`prior != short`), so two DISTINCT aliases landing on the IDENTICAL short compared equal and passed. Key by the owning inner alias instead, which catches exact duplicates as well as the case-only and fitted-form variants it already caught. Enrichment has a related guard, but it does not cover this: its `_occupied_shorts` map is populated from dimensions and checked only against MEASURES, so dimension-vs-dimension slips through, and its local flattening helper applies no length fitting, so two shorts that differ before fitting and collide after it pass there too. A root column named `SandboxSubscription__SandboxCustomer__SandboxConsumer__name` alongside the deep dimension that flattens to the same string reaches the emission boundary undetected — column names permit `__`, so this is reachable. 2/3. BigQuery and T-SQL `decode_result_keys` pre-decoded row keys in a dict comprehension and only then passed the result to `_rekey_row`. Two keys normalizing to the same name collapsed in that intermediate dict, with one value silently dropped, before the duplicate check ever ran — the exact failure mode this PR exists to prevent. `_rekey_row` now takes an optional `fallback` decoder so mapping lookup, bijection fallback, and duplicate detection all happen in one pass. Also fixes 12 SonarQube findings in the new tests (3 of them bug-class `python:S5863`, which failed the quality gate on `new_reliability_rating`): tautological `f(x) == f(x)` assertions replaced with pinned expected values, composite `assert a and b` split, and `get_dialect(...)` hoisted out of `pytest.raises` blocks so only one call inside can throw. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/query_engine.py | 20 +++++--- slayer/sql/dialects/base.py | 22 ++++++-- slayer/sql/dialects/bigquery.py | 8 +-- slayer/sql/dialects/tsql.py | 8 +-- tests/dialects/test_identifier_fit.py | 47 ++++++++++++++--- tests/test_dev1756_identifier_length.py | 67 ++++++++++++++++++++++--- 6 files changed, 138 insertions(+), 34 deletions(-) diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index 108911d1..ef2c6bba 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -3314,16 +3314,20 @@ def _alias_to_short(alias: str) -> str: def _short_sql(short: str) -> str: return exp.Identifier(this=short, quoted=True).sql(dialect=dialect) - # DEV-1756: the shorts share one output-column namespace. Two that - # differ only by case would still collide were they ever emitted bare, - # and two that fit to the same string always collide, so validate the - # whole allocation before emitting. + # DEV-1756: the shorts share one output-column namespace, and each also + # becomes a ``Column.name`` on the virtual model. Validate the whole + # allocation keyed by the OWNING inner alias, which catches three + # things at once: two shorts that fit to the same string, two that + # differ only by case, and — the case ``_alias_to_short`` cannot + # prevent — two distinct aliases landing on the identical short because + # a user-declared cross-model ``name`` bypassed the flattening above + # and happened to match another column's canonical flat. short_owner: dict[str, str] = {} - for _, short, _, _, _, _ in column_map: - prior = short_owner.setdefault(short.casefold(), short) - if prior != short: + for alias, short, _, _, _, _ in column_map: + prior_alias = short_owner.setdefault(short.casefold(), alias) + if prior_alias != alias: raise IdentifierCollisionError( - first=prior, second=short, emitted=short.casefold(), + first=prior_alias, second=alias, emitted=short, dialect=dialect, limit=get_dialect(dialect).max_identifier_bytes, namespace="query-backed model column", diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 6e721985..cd79daa2 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -573,13 +573,29 @@ def decode_alias_map(self, aliases: Sequence[str]) -> dict[str, str]: return out def _rekey_row( - self, row: dict[str, Any], mapping: dict[str, str], + self, + row: dict[str, Any], + mapping: dict[str, str], + *, + fallback: Callable[[str], str] | None = None, ) -> dict[str, Any]: """Apply ``mapping`` to one row's keys, refusing to let two keys - collapse onto one (which would silently drop a column's values).""" + collapse onto one (which would silently drop a column's values). + + ``fallback`` decodes keys absent from ``mapping`` — BigQuery and T-SQL + pass their ``___`` -> ``.`` bijection. It must be applied HERE rather + than in a separate dict comprehension upstream: pre-decoding into a + dict would let two keys collapse before this check ever ran, which is + precisely the silent-column-loss this class exists to prevent. + """ out: dict[str, Any] = {} for key, value in row.items(): - decoded = mapping.get(key, key) + if key in mapping: + decoded = mapping[key] + elif fallback is not None: + decoded = fallback(key) + else: + decoded = key if decoded in out: raise IdentifierCollisionError( first=key, second=decoded, emitted=decoded, diff --git a/slayer/sql/dialects/bigquery.py b/slayer/sql/dialects/bigquery.py index 33e3913d..843e5b39 100644 --- a/slayer/sql/dialects/bigquery.py +++ b/slayer/sql/dialects/bigquery.py @@ -187,13 +187,13 @@ def decode_result_keys( DEV-1756: keys produced by a length-fitted alias are not recoverable from the key alone, so the ``emitted -> canonical`` map is consulted first; anything outside it falls back to the pure ``___`` -> ``.`` - bijection, preserving today's behaviour for short aliases. + bijection, preserving today's behaviour for short aliases. Both steps + happen inside ``_rekey_row`` in ONE pass — pre-decoding into a dict + first would let two keys collapse before the duplicate check ran. """ - mapping = self.decode_alias_map(aliases) return [ self._rekey_row( - {decode_alias(k) if k not in mapping else k: v for k, v in row.items()}, - mapping, + row, self.decode_alias_map(aliases), fallback=decode_alias, ) for row in rows ] diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index 11c7b5a6..6ad07066 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -394,13 +394,13 @@ def decode_result_keys( DEV-1756: keys produced by a length-fitted alias are not recoverable from the key alone, so the ``emitted -> canonical`` map is consulted first; anything outside it falls back to the pure ``___`` -> ``.`` - bijection, preserving today's behaviour for short aliases. + bijection, preserving today's behaviour for short aliases. Both steps + happen inside ``_rekey_row`` in ONE pass — pre-decoding into a dict + first would let two keys collapse before the duplicate check ran. """ - mapping = self.decode_alias_map(aliases) return [ self._rekey_row( - {decode_alias(k) if k not in mapping else k: v for k, v in row.items()}, - mapping, + row, self.decode_alias_map(aliases), fallback=decode_alias, ) for row in rows ] diff --git a/tests/dialects/test_identifier_fit.py b/tests/dialects/test_identifier_fit.py index 8b70d707..90c88bc0 100644 --- a/tests/dialects/test_identifier_fit.py +++ b/tests/dialects/test_identifier_fit.py @@ -105,8 +105,17 @@ def test_output_within_limit_bytes(self, limit: int) -> None: name = "Sandbox" * 80 # 560 bytes — over every configured limit assert _nbytes(fit_identifier(name, limit=limit)) <= limit - def test_deterministic_across_calls(self) -> None: - assert fit_identifier(LONG_EMAIL, limit=63) == fit_identifier(LONG_EMAIL, limit=63) + def test_output_is_pinned_exactly(self) -> None: + """Pin the whole result, not `f(x) == f(x)` — which proves nothing + about a pure function. Also documents the head/tail split concretely: + 27 head bytes, the 10-byte marker, 26 tail bytes.""" + expected = ( + "SandboxInvoiceV2.SandboxSub" + f"_{hashlib.sha256(LONG_EMAIL.encode()).hexdigest()[:HASH_LEN]}_" + "omer.SandboxConsumer.email" + ) + assert fit_identifier(LONG_EMAIL, limit=63) == expected + assert _nbytes(expected) == 63 def test_marker_is_exactly_sha256_of_the_full_original(self) -> None: """Pin the digest's ALGORITHM, POSITION and INPUT — not merely that @@ -145,7 +154,8 @@ def test_repro_siblings_differ_outside_the_hash(self) -> None: a = fit_identifier(LONG_NAME, limit=63) b = fit_identifier(LONG_EMAIL, limit=63) assert a != b - assert a.endswith("name") and b.endswith("email") + assert a.endswith("name") + assert b.endswith("email") def test_distinct_inputs_sharing_63_byte_prefix_produce_distinct_outputs(self) -> None: a = fit_identifier(LONG_NAME, limit=63) @@ -458,9 +468,11 @@ def test_digest_collision_raises(self, monkeypatch: pytest.MonkeyPatch) -> None: import slayer.sql.dialects._identifier_fit as fitmod monkeypatch.setattr(fitmod, "_digest", lambda name: "deadbeef") + pg = get_dialect("postgres") with pytest.raises(IdentifierCollisionError) as exc: - get_dialect("postgres").alias_rewrite_map([TWIN_A, TWIN_B]) - assert TWIN_A in str(exc.value) and TWIN_B in str(exc.value) + pg.alias_rewrite_map([TWIN_A, TWIN_B]) + assert TWIN_A in str(exc.value) + assert TWIN_B in str(exc.value) def test_shortened_form_equal_to_an_existing_short_alias_raises(self) -> None: """The guard must consider IDENTITY entries too. A short alias whose @@ -476,9 +488,11 @@ def test_error_names_the_dialect_and_limit(self, monkeypatch: pytest.MonkeyPatch import slayer.sql.dialects._identifier_fit as fitmod monkeypatch.setattr(fitmod, "_digest", lambda name: "deadbeef") + pg = get_dialect("postgres") with pytest.raises(IdentifierCollisionError) as exc: - get_dialect("postgres").alias_rewrite_map([TWIN_A, TWIN_B]) - assert "postgres" in str(exc.value) and "63" in str(exc.value) + pg.alias_rewrite_map([TWIN_A, TWIN_B]) + assert "postgres" in str(exc.value) + assert "63" in str(exc.value) def test_is_a_slayer_error_and_value_error(self) -> None: from slayer.core.errors import SlayerError @@ -500,3 +514,22 @@ def test_two_keys_decoding_to_one_canonical_raises(self) -> None: rows = [{pg.fit_alias(LONG_EMAIL): 1, LONG_EMAIL: 2}] with pytest.raises(IdentifierCollisionError): pg.decode_result_keys(rows, aliases=[LONG_EMAIL]) + + @pytest.mark.parametrize("dialect", ["bigquery", "tsql"]) + def test_mangle_fallback_collision_raises(self, dialect: str) -> None: + """The dot-mangling dialects decode unmapped keys through + ``decode_alias``. That fallback must run INSIDE the duplicate check — + pre-decoding into a dict first would let ``orders___status`` and + ``orders.status`` collapse onto one key with one value silently + dropped, before any collision could be observed.""" + d = get_dialect(dialect) + rows = [{"orders___status": 1, "orders.status": 2}] + with pytest.raises(IdentifierCollisionError): + d.decode_result_keys(rows, aliases=[]) + + @pytest.mark.parametrize("dialect", ["bigquery", "tsql"]) + def test_mangle_fallback_preserves_every_value(self, dialect: str) -> None: + """No key is dropped when nothing collides.""" + d = get_dialect(dialect) + got = d.decode_result_keys([{"a___b": 1, "c___d": 2}], aliases=[]) + assert got == [{"a.b": 1, "c.d": 2}] diff --git a/tests/test_dev1756_identifier_length.py b/tests/test_dev1756_identifier_length.py index c7b9b699..5dbf3c5c 100644 --- a/tests/test_dev1756_identifier_length.py +++ b/tests/test_dev1756_identifier_length.py @@ -122,7 +122,11 @@ # --------------------------------------------------------------------------- -def _chain_models(*, case_colliding_columns: bool = False) -> list[SlayerModel]: +def _chain_models( + *, + case_colliding_columns: bool = False, + decoy_root_column: str | None = None, +) -> list[SlayerModel]: consumer_columns = [ Column(name="id", sql="id", type=DataType.INT, primary_key=True), Column(name="name", sql="name", type=DataType.TEXT), @@ -141,6 +145,12 @@ def _chain_models(*, case_colliding_columns: bool = False) -> list[SlayerModel]: Column(name="status", sql="status", type=DataType.TEXT), Column(name="totalAmount", sql="total_amount", type=DataType.DOUBLE), Column(name="subscription_id", sql="subscription_id", type=DataType.INT), + *([ + # A root column named exactly what a deep join path flattens + # to. Column names permit ``__`` (the query-backed carve-out), + # so this is reachable, not contrived. + Column(name=decoy_root_column, sql="decoy", type=DataType.TEXT), + ] if decoy_root_column else []), ], joins=[ModelJoin( target_model="SandboxSubscription", join_pairs=[["subscription_id", "id"]], @@ -340,8 +350,10 @@ async def test_canonical_aliases_are_over_the_limit(self, chain) -> None: engine, model = chain enriched = await engine._enrich(query=_repro_query(), model=model) aliases = public_projection_aliases(enriched) - assert LONG_NAME in aliases and LONG_EMAIL in aliases - assert _nbytes(LONG_NAME) == 73 and _nbytes(LONG_EMAIL) == 74 + assert LONG_NAME in aliases + assert LONG_EMAIL in aliases + assert _nbytes(LONG_NAME) == 73 + assert _nbytes(LONG_EMAIL) == 74 def test_the_two_aliases_share_a_63_byte_prefix(self) -> None: assert LONG_NAME.encode()[:63] == LONG_EMAIL.encode()[:63] @@ -377,7 +389,8 @@ async def test_outer_wrap_uses_the_same_token_everywhere(self, chain) -> None: outer_names = {n for n, _ in _projection_aliases(selects[0])} inner_names = {n for n, _ in _projection_aliases(selects[1])} fitted = get_dialect("postgres").fit_alias(LONG_EMAIL) - assert fitted in outer_names and fitted in inner_names + assert fitted in outer_names + assert fitted in inner_names _assert_within_limit(sql, 63) _assert_no_namespace_collision(sql) _assert_order_by_refs_resolve(sql) @@ -554,10 +567,17 @@ async def test_includes_hidden_order_by_hoist(self, chain) -> None: hidden = [a for a in all_projection_aliases(enriched) if a not in public] assert any("totalAmount_avg" in a for a in hidden), hidden - async def test_is_deterministic(self, chain) -> None: + async def test_is_stable_across_calls(self, chain) -> None: + """Order must not depend on set iteration — the rewrite map is derived + from this list.""" engine, model = chain enriched = await engine._enrich(query=_repro_query(), model=model) - assert all_projection_aliases(enriched) == all_projection_aliases(enriched) + first = all_projection_aliases(enriched) + assert first == [ + LONG_NAME, LONG_EMAIL, "SandboxInvoiceV2.status", + "SandboxInvoiceV2.totalAmount_sum", "SandboxInvoiceV2._count", + ] + assert all_projection_aliases(enriched) == first # =========================================================================== @@ -634,7 +654,9 @@ def test_cte_allocator_is_idempotent_for_the_same_owner(self) -> None: """Several code paths re-derive a CTE's name in order to reference it; that must not read as a collision.""" gen = SQLGenerator(dialect="postgres") - assert gen._cte_name("_cm_", TWIN_A) == gen._cte_name("_cm_", TWIN_A) + first = gen._cte_name("_cm_", TWIN_A) + second = gen._cte_name("_cm_", TWIN_A) # hits the memo, must not raise + assert second == first def test_cte_allocator_resets_per_statement(self) -> None: """One generator instance generates many statements; allocation is @@ -728,7 +750,8 @@ async def test_short_aliases_are_quoted(self, chain) -> None: for proj in select.expressions: assert isinstance(proj, exp.Alias), f"{proj.sql()} is not an aliased projection" ident = proj.args.get("alias") - assert isinstance(ident, exp.Identifier) and ident.quoted, ( + assert isinstance(ident, exp.Identifier), f"{ident} is not an Identifier" + assert ident.quoted, ( f"short alias {ident} must be dialect-quoted\n{vm.sql}" ) @@ -765,6 +788,34 @@ async def test_case_colliding_shorts_raise(self, tmp_path) -> None: with pytest.raises(IdentifierCollisionError): await engine._query_as_model(inner_query=query) + async def test_two_dimensions_landing_on_one_short_raise(self, tmp_path) -> None: + """Two DIMENSIONS whose shorts are exactly equal must be caught. + + `SandboxSubscription.SandboxCustomer.SandboxConsumer.name` flattens to + `SandboxSubscription__SandboxCustomer__SandboxConsumer__name`, which a + root column may also be named literally (column names permit `__`). + Both become `Column.name` on the virtual model. + + Enrichment's own guard does NOT cover this: `_occupied_shorts` is + populated from dimensions but only checked against MEASURES, so + dimension-vs-dimension slips through. It also flattens without length + fitting, so two shorts that differ before fitting and collide after it + would pass there too. Hence the check at the emission boundary — and it + has to key on the owning alias, since comparing the two shorts to each + other finds them equal and waves the pair through. + """ + flat = "SandboxSubscription__SandboxCustomer__SandboxConsumer__name" + engine, _ = await _build_engine(tmp_path, decoy_root_column=flat) + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name=f"{DEEP}.name"), ColumnRef(name=flat)], + measures=[{"formula": "*:count"}], + ) + with pytest.raises(IdentifierCollisionError) as exc: + await engine._query_as_model(inner_query=query) + assert "query-backed model column" in str(exc.value) + assert flat in str(exc.value) + async def test_nested_dag_two_levels_agree(self, chain) -> None: """Two stages: stage 2 references stage 1's virtual-model columns.""" engine, _ = chain From 461a73b849c4b5731133a6923fd81f8b8d4fc378 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 16:05:52 +0200 Subject: [PATCH 03/10] perf(DEV-1756): build the decode map once per call, not once per row Follow-up to the previous commit: moving `decode_alias_map(aliases)` into the list comprehension rebuilt it for every result row, turning decoding from O(aliases + rows x columns) into O(rows x aliases + rows x columns). The map is a pure function of `aliases`, so hoist it back out of the loop. Codex review finding on PR #289. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/dialects/bigquery.py | 5 ++--- slayer/sql/dialects/tsql.py | 5 ++--- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/slayer/sql/dialects/bigquery.py b/slayer/sql/dialects/bigquery.py index 843e5b39..0373f13c 100644 --- a/slayer/sql/dialects/bigquery.py +++ b/slayer/sql/dialects/bigquery.py @@ -191,10 +191,9 @@ def decode_result_keys( happen inside ``_rekey_row`` in ONE pass — pre-decoding into a dict first would let two keys collapse before the duplicate check ran. """ + mapping = self.decode_alias_map(aliases) return [ - self._rekey_row( - row, self.decode_alias_map(aliases), fallback=decode_alias, - ) + self._rekey_row(row, mapping, fallback=decode_alias) for row in rows ] diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index 6ad07066..ac05cd64 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -398,9 +398,8 @@ def decode_result_keys( happen inside ``_rekey_row`` in ONE pass — pre-decoding into a dict first would let two keys collapse before the duplicate check ran. """ + mapping = self.decode_alias_map(aliases) return [ - self._rekey_row( - row, self.decode_alias_map(aliases), fallback=decode_alias, - ) + self._rekey_row(row, mapping, fallback=decode_alias) for row in rows ] From e70e7e0d5fa5086e788e411f9c8f9396b8f530ae Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 18:27:00 +0200 Subject: [PATCH 04/10] fix(DEV-1756): match short-alias quoting to the downstream reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review findings on the identifier-length work, plus two doc fixes. 1. `_query_as_model` quoted its short alias UNCONDITIONALLY, which fixed the Postgres mixed-case bug by creating the mirror-image bug on upper-folding backends: the wrapper defined a case-sensitive `"status"` while the downstream `Column(sql="status")` reference stayed bare (only `_quote_mixed_case_identifiers` quotes it, and only when it contains an uppercase letter) and resolved as `STATUS` on Snowflake/Oracle. The contract is AGREEMENT between the two sides, not quoting, so `_short_sql` now mirrors the reference-side policy: quote iff mixed-case or reserved. Three tests bracket it — the agreement test fails against both the pre-PR version and the always-quote version. 2. Caller-supplied virtual-model shorts bypassed the length fitting. `m.name` / `t.name` / `e.name` and a user-declared cross-model `name` went into `column_map` verbatim, so two over-limit names sharing a 63-byte prefix were emitted as `AS "<64+ bytes>"` and truncated onto one column by Postgres — invisible to the `short_owner` check, which compares Python strings. New `_fit_short` covers all four sites plus the DEV-1449 `agg_shorts` breadcrumb, so it cannot drift from `column_map`. 3. `substitute_quoted`'s "cannot reach into a string literal" was an overclaim. Being keyed to an exact alias set bounds the exposure to a literal holding the exact quoted spelling of one of the same query's over-limit aliases; it does not eliminate it. Docstrings corrected — no code change, since the BigQuery/T-SQL dot-mangle regexes already run over this SQL with strictly wider exposure. 4. The `docs/database-support.md` worked example was fabricated: a 53-byte input, under the limit, paired with output `fit_identifier` never produces. Replaced with real 68 -> 63 output. Co-Authored-By: Claude Opus 5 (1M context) --- DECISIONS.md | 2 +- docs/database-support.md | 4 +- slayer/engine/query_engine.py | 62 ++++++++++++---- slayer/sql/dialects/_identifier_fit.py | 14 +++- slayer/sql/dialects/base.py | 3 +- tests/test_dev1756_identifier_length.py | 96 +++++++++++++++++++++++-- tests/test_query_backed_models.py | 17 ++--- 7 files changed, 163 insertions(+), 35 deletions(-) diff --git a/DECISIONS.md b/DECISIONS.md index db35f3be..bd70a504 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -70,4 +70,4 @@ implementation detail. Include issue refs when known. - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. - 2026-08-05 — Ingestion sees views, survives unmodellable names, and stops being silent (DEV-1741). **Views**: `list_ingestable_objects` replaces the bare `get_table_names()` at every introspection site, adding `get_view_names` + `get_materialized_view_names` behind a `NotImplementedError`/`Exception` guard (the base `Inspector` RAISES for matviews on unsupporting dialects rather than returning `[]`), de-duplicated first-classification-wins because some dialects return views from `get_table_names()`, in a deterministic tables→views→matviews order that the name-collision policy depends on. Ingested by default — dbt materializes staging models as views, so opt-in would have left the reported failure in place for a fresh install — with `--no-views` on both `slayer ingest` and `datasources create --ingest`. The drift side (`_live_schema_for_datasource`) and the MCP listing (`_fetch_tables`) take **no flag and are unconditional**: that map is only ever a lookup target (`validate_datasource` iterates the *persisted* models, `available_in_ds` derives from them), so views there cannot manufacture a model or a drift entry — what they fix is a pre-existing **data-loss** bug where a hand-authored model whose `sql_table` named a view resolved to `live_table=None` → `WholeModelDelete` → deleted by `validate-models --force-clean`. Gating that on `--no-views` would re-arm it for exactly the users who opted out. **Names**: model names can't contain `__` (six modules — generator/enrichment/column_expansion/column_dependency/schema_drift/osi — `split("__")` an alias back into a join path, so a model named `a__b` is read as the alias for `a→b` and yields a silently wrong query, not a crash), but `sql_table` can, so a dlt child table `reports__patient__drug` is modelled as `reports_patient_drug` with `sql_table` verbatim. Sanitizer is `re.sub(r"_{2,}", "_")`, NOT `replace("__","_")` — `str.replace` is non-overlapping, so `a___b`→`a__b` would still fail validation. Collisions reserve every unsanitized name first (a real `a_b` always beats a sanitized `a__b`, order-independently) and **skip** rather than suffix, since suffixes shift as the object set changes and would orphan models and churn drift. A per-object `try/except` backstops everything else (`.`/`:`/`/`/`\` names, bad column names, per-object introspection failures); the FK-collection loop and `_get_fk_relationships` are guarded too because they run BEFORE that isolation and would otherwise still kill the run. **Reporting**: skips travel in a new `skipped` list, deliberately NOT folded into `errors` — `slayer ingest` exits 1 on either (we declined a perfectly valid object; `--exclude` is the documented remedy) but `POST /ingest` keeps 422 for `errors` only, because a permanent 422 aimed at a machine that can't act on the hint buries a successful partial ingest behind an error status. An empty scan prints the available schemas and exits 1 (the reporter listed the missing exit code as part of the defect), gated on `objects` not `additions` so a healthy no-op re-ingest stays quiet; `datasources create --ingest` on an empty DB still exits 0, since creating the datasource is that command's job and it succeeded. `in_scope_table_names` switched from model names to `_bare_table_name(sql_table)` — it is compared against table names in `_scoped_models_for_validation`, so the old keying silently dropped from validation scope any model whose name differs from its table (every sanitized model, and already every dbt/OSI hidden model passing `model_name=`). **`source_kind`** (`table`/`view`/`materialized_view`/`None`=unknown) persists on `SlayerModel`, v7→v8 with a no-op migration (mandatory: `migrate()` raises `RuntimeError` on an unregistered step); `None` for pre-v8, hand-authored and sql/query-backed models is the honest value, not a guess. It is a deliberate **exception to the additive-merge contract** — refreshed, not preserved — because it describes the live object rather than user intent, and the transition it exists to capture (dbt `+materialized: table`) usually changes no columns at all; the refresh therefore has to be made in three places (the early return, the `model_copy(update=...)`, and the save gate in `_process_one_table`), since editing only the update dict computes a corrected model and throws it away. A `None` from a non-classifying path never erases a known value. Docs-only fix for the advertised `motley-slayer[duckdb]` extra, which does not exist: `duckdb`/`duckdb-engine` are unconditional core deps (the Postgres facade imports duckdb at top level on every CLI invocation), and adding them under `[tool.poetry.extras]` would *gate* rather than alias them, breaking bare `pip install motley-slayer` → `datasources create demo`. -- 2026-08-06 — Dialect-aware identifier-length fitting (DEV-1756): every dialect declares a conservative universal budget as `SqlDialect.max_identifier_bytes` (postgres 63, mysql 64, redshift 127, oracle/tsql 128, snowflake 255, duckdb 256, bigquery 300, `None` = unbounded for sqlite/clickhouse/trino/presto/databricks/spark), and an over-limit identifier is shortened at emission to `__` via `slayer/sql/dialects/_identifier_fit.py`. Postgres is the binding case and the reason this is a correctness fix rather than cosmetics: it truncates over-length identifiers **silently** (a NOTICE, never an error), so two 3-hop aliases sharing a 63-byte prefix either blow up as `AmbiguousColumnError` under the DEV-1444 outer wrap or — with no sibling to collide with — quietly return the column under a name the engine never looks up. Shorten **only when over the limit**, never uniformly: `decode_result_keys` restores the canonical dotted alias, so the dialect-dependence the issue worried about is invisible to consumers and the 99% case keeps byte-identical SQL (pinned by pre-change goldens, not merely by idempotence). Both ends of the alias are kept because the reported colliding pair differs ONLY in its final segment — a head-only truncation would render the two indistinguishable in `dry_run` output. The digest is sha256 of the FULL original, which is what makes `fit_identifier` a pure function of the name and lets the read side rebuild the emitted→canonical map by re-running it, with no map threaded through generation (the alternative the issue sketched). Write side is an EXACT-match replacement over the query's own alias set (`all_projection_aliases`, unfiltered — hidden ORDER-BY hoists and `_inner_*`/`_ft*`/`_ts*` entries are projected in the inner SELECT and truncate identically), never a length regex over arbitrary SQL, so a long quoted-looking span inside a string literal can never be corrupted; substitution is two-phase (canonical→sentinel→final) as defence-in-depth, though today's key set (over-limit) and value set (within-limit) are provably disjoint. BigQuery/T-SQL compose by running their existing dot-mangle regex AFTER the base length pass: an under-limit alias makes the length pass a genuine no-op so their output is unchanged, and an over-limit one arrives still-dotted and gets mangled by the same regex — no double-encoding — with the budget sized against `encode_alias` so the post-mangle form still fits. Scope is the three OUTPUT-name surfaces: projection aliases, CTE names (`_cte_name_from_alias` fits the whole result, prefix included, allocated through a per-statement collision-checked `SQLGenerator._cte_name`), and `_query_as_model` virtual-model shorts. Join-path TABLE aliases are deferred to DEV-1743, whose plan already owns the `__` path-alias allocator; their failure mode (silent wrong joins) is worse but the fix requires decoupling `EnrichedDimension.model_name` from the emitted qualifier across ~8 sites that split it back on `__`. Collisions raise `IdentifierCollisionError` rather than emitting ambiguous SQL — the check covers identity entries too, since an already-short alias equal to another's fitted form is a duplicate no hash width can prevent. Fixed alongside, in the same code path: `_query_as_model` now ALWAYS dialect-quotes its short alias instead of only for reserved words — emitted bare, a mixed-case short was case-folded by Postgres while the outer stage referenced it quoted, making any query-backed model with a mixed-case join path unqueryable (`UndefinedColumnError`, reproduced on a live server); quoting also takes these names out of the case-folding namespace, so their uniqueness check is the only one that needs to be case-insensitive. +- 2026-08-06 — Dialect-aware identifier-length fitting (DEV-1756): every dialect declares a conservative universal budget as `SqlDialect.max_identifier_bytes` (postgres 63, mysql 64, redshift 127, oracle/tsql 128, snowflake 255, duckdb 256, bigquery 300, `None` = unbounded for sqlite/clickhouse/trino/presto/databricks/spark), and an over-limit identifier is shortened at emission to `__` via `slayer/sql/dialects/_identifier_fit.py`. Postgres is the binding case and the reason this is a correctness fix rather than cosmetics: it truncates over-length identifiers **silently** (a NOTICE, never an error), so two 3-hop aliases sharing a 63-byte prefix either blow up as `AmbiguousColumnError` under the DEV-1444 outer wrap or — with no sibling to collide with — quietly return the column under a name the engine never looks up. Shorten **only when over the limit**, never uniformly: `decode_result_keys` restores the canonical dotted alias, so the dialect-dependence the issue worried about is invisible to consumers and the 99% case keeps byte-identical SQL (pinned by pre-change goldens, not merely by idempotence). Both ends of the alias are kept because the reported colliding pair differs ONLY in its final segment — a head-only truncation would render the two indistinguishable in `dry_run` output. The digest is sha256 of the FULL original, which is what makes `fit_identifier` a pure function of the name and lets the read side rebuild the emitted→canonical map by re-running it, with no map threaded through generation (the alternative the issue sketched). Write side is an EXACT-match replacement over the query's own alias set (`all_projection_aliases`, unfiltered — hidden ORDER-BY hoists and `_inner_*`/`_ft*`/`_ts*` entries are projected in the inner SELECT and truncate identically), never a length regex over arbitrary SQL, so a long quoted-looking span inside a string literal can never be corrupted; substitution is two-phase (canonical→sentinel→final) as defence-in-depth, though today's key set (over-limit) and value set (within-limit) are provably disjoint. BigQuery/T-SQL compose by running their existing dot-mangle regex AFTER the base length pass: an under-limit alias makes the length pass a genuine no-op so their output is unchanged, and an over-limit one arrives still-dotted and gets mangled by the same regex — no double-encoding — with the budget sized against `encode_alias` so the post-mangle form still fits. Scope is the three OUTPUT-name surfaces: projection aliases, CTE names (`_cte_name_from_alias` fits the whole result, prefix included, allocated through a per-statement collision-checked `SQLGenerator._cte_name`), and `_query_as_model` virtual-model shorts. Join-path TABLE aliases are deferred to DEV-1743, whose plan already owns the `__` path-alias allocator; their failure mode (silent wrong joins) is worse but the fix requires decoupling `EnrichedDimension.model_name` from the emitted qualifier across ~8 sites that split it back on `__`. Collisions raise `IdentifierCollisionError` rather than emitting ambiguous SQL — the check covers identity entries too, since an already-short alias equal to another's fitted form is a duplicate no hash width can prevent. Fixed alongside, in the same code path: `_query_as_model` dialect-quotes its short alias when it is mixed-case as well as when it is a reserved word. Emitted bare, a mixed-case short was case-folded by Postgres while the outer stage referenced it quoted (DEV-1645 quotes mixed-case `Column.sql` leaves), making any query-backed model with a mixed-case join path unqueryable (`UndefinedColumnError`, reproduced on a live server). The contract is AGREEMENT between the two sides, not quoting: the policy deliberately mirrors `_quote_mixed_case_identifiers` + `prequote_reserved_identifiers`, which is what the downstream `Column(sql=short)` reference is emitted under. Quoting UNCONDITIONALLY was tried first and is wrong in the mirror direction — it defines a case-sensitive `"status"` on upper-folding backends (Snowflake, Oracle) while the bare reference still resolves as `STATUS`, so it trades a Postgres bug for a Snowflake one. Because an all-lowercase short therefore stays bare and still shares a case-folded namespace, the short-uniqueness check remains case-insensitive. diff --git a/docs/database-support.md b/docs/database-support.md index 671011d5..a33be99c 100644 --- a/docs/database-support.md +++ b/docs/database-support.md @@ -52,8 +52,8 @@ form that keeps both the root model and the column name readable, with an 8-hex-character SHA-256 of the full original in between: ``` -orders.customers.regions.districts.neighbourhood_name (over 63 bytes) - -> orders.customers.region_4f2a91c7_ricts.neighbourhood_name +orders.customers.regions.districts.neighbourhoods.neighbourhood_name (68 bytes) + -> orders.customers.regions.di_e6932600_urhoods.neighbourhood_name (63 bytes) ``` Two properties matter for consumers: diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index ef2c6bba..b5ac5074 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -81,6 +81,7 @@ from slayer.sql import engine_factory from slayer.sql.engine_factory import _runtime_fingerprint from slayer.sql.generator import SQLGenerator +from slayer.sql.reserved_keywords import SLAYER_RESERVED_KEYWORDS from slayer.sql.session_policy import ScopedTable, apply_session_policy from slayer.storage.base import StorageBackend @@ -3239,9 +3240,23 @@ def _alias_to_short(alias: str) -> str: # Strip source model prefix stripped = alias.split(".", 1)[-1] if "." in alias else alias # Replace remaining dots with __ to encode the original join path + return _fit_short(stripped.replace(".", "__")) + + def _fit_short(name: str) -> str: + """DEV-1756: bound a virtual-model short to the dialect's budget. + + Every short reaches the SQL as an output-column alias AND becomes a + ``Column.name`` downstream, so they all need this — not just the + ``_alias_to_short`` flattened ones. A measure/transform/expression + ``name``, or a user-declared cross-model rename, is supplied by the + caller and can be arbitrarily long; two such names sharing a + 63-byte prefix are silently truncated onto one column by Postgres, + and the ``short_owner`` check below cannot see it because they + differ as Python strings. Identity under the limit, so ordinary + names are untouched. + """ return fit_identifier( - stripped.replace(".", "__"), - limit=get_dialect(dialect).max_identifier_bytes, + name, limit=get_dialect(dialect).max_identifier_bytes, ) # (inner_alias, short_name, data_type, label, description, format) @@ -3265,14 +3280,14 @@ def _alias_to_short(alias: str) -> str: measure_name=src_name, aggregation=m.aggregation, ) - column_map.append((m.alias, m.name, DataType.DOUBLE, label, desc, fmt)) + column_map.append((m.alias, _fit_short(m.name), DataType.DOUBLE, label, desc, fmt)) for t in enriched.transforms: column_map.append( - (t.alias, t.name, DataType.DOUBLE, t.label, None, NumberFormat(type=NumberFormatType.FLOAT)) + (t.alias, _fit_short(t.name), DataType.DOUBLE, t.label, None, NumberFormat(type=NumberFormatType.FLOAT)) ) for e in enriched.expressions: column_map.append( - (e.alias, e.name, DataType.DOUBLE, e.label, None, NumberFormat(type=NumberFormatType.FLOAT)) + (e.alias, _fit_short(e.name), DataType.DOUBLE, e.label, None, NumberFormat(type=NumberFormatType.FLOAT)) ) for cm in enriched.cross_model_measures: # DEV-1448: when the user supplied an explicit ``name``, cm.name is @@ -3291,7 +3306,10 @@ def _alias_to_short(alias: str) -> str: # leak into the virtual model's column set. Only user-declared # renames qualify for the bare-name short. if cm.user_declared and cm.name and "." not in cm.name: - short = cm.name + # DEV-1756: still length-fitted. A rename the backend would + # truncate is not the name the user gets either way; fitting at + # least makes it deterministic and collision-checked. + short = _fit_short(cm.name) else: short = _alias_to_short(cm.alias) column_map.append((cm.alias, short, DataType.DOUBLE, cm.label, None, cm.format)) @@ -3304,15 +3322,27 @@ def _alias_to_short(alias: str) -> str: # would either fail to parse (MySQL) or reference an alias the # mangled inner subquery doesn't expose (T-SQL). # DEV-1686: the inner ``alias`` is always dialect-quoted; the ``short`` - # output alias must be too. It was originally quoted only for reserved - # words, but a BARE mixed-case short is case-folded by Postgres while - # the outer stage references it quoted (DEV-1645 quotes mixed-case - # ``Column.sql`` leaves) — so any query-backed model with a mixed-case - # join path failed with ``UndefinedColumnError``. Quoting always fixes - # that and, per DEV-1756, keeps these names out of the case-folding - # namespace entirely. + # output alias is quoted under exactly the policy the DOWNSTREAM + # reference uses, because the two have to agree. + # + # A downstream ``Column(sql=short)`` is parsed by the generator, where + # ``_quote_mixed_case_identifiers`` (DEV-1645) quotes a leaf iff it + # contains an uppercase letter, and ``prequote_reserved_identifiers`` + # (DEV-1686) quotes reserved words. So: + # * reserved / mixed-case -> quoted BOTH sides. This is the DEV-1756 + # fix: emitted bare, a mixed-case short was case-folded by Postgres + # while the outer stage referenced it quoted, making any + # query-backed model with a mixed-case join path or column + # unqueryable (``UndefinedColumnError``, seen on a live server). + # * all-lowercase -> bare BOTH sides, so it folds consistently. + # Quoting unconditionally instead breaks the lowercase case on + # UPPER-folding backends (Snowflake, Oracle): the wrapper would define a + # case-sensitive ``"status"`` while the bare reference resolves as + # ``STATUS``. def _short_sql(short: str) -> str: - return exp.Identifier(this=short, quoted=True).sql(dialect=dialect) + if short.lower() in SLAYER_RESERVED_KEYWORDS or any(c.isupper() for c in short): + return exp.Identifier(this=short, quoted=True).sql(dialect=dialect) + return short # DEV-1756: the shorts share one output-column namespace, and each also # becomes a ``Column.name`` on the virtual model. Validate the whole @@ -3375,7 +3405,9 @@ def _short_sql(short: str) -> str: agg_shorts.add(_alias_to_short(cm.alias)) for m in enriched.measures: if m.from_cross_model_intercept: - agg_shorts.add(m.name) + # DEV-1756: same fitting the ``column_map`` entry got, or the + # breadcrumb would name a column the virtual model never has. + agg_shorts.add(_fit_short(m.name)) # DEV-1449: record the lineage breadcrumb so outer-stage dotted-ref # lookup can strip the right ancestor prefix and find the flat diff --git a/slayer/sql/dialects/_identifier_fit.py b/slayer/sql/dialects/_identifier_fit.py index 2fdac46b..d1727bff 100644 --- a/slayer/sql/dialects/_identifier_fit.py +++ b/slayer/sql/dialects/_identifier_fit.py @@ -19,8 +19,10 @@ :func:`substitute_quoted` Rewrites quoted identifier tokens in emitted SQL. Driven by an exact - canonical->emitted map rather than a length regex, so it can never reach - into a string literal that happens to contain a long quoted-looking span. + canonical->emitted map rather than a length regex, so — unlike a regex over + arbitrary quoted spans — the only text it can reach inside a string literal + is the exact quoted spelling of one of *this query's own* over-limit + aliases. See its docstring for the residual case. Sibling of :mod:`slayer.sql.dialects._alias_mangle` (the BigQuery/T-SQL dotted alias codec) and composes with it: those dialects size against ``encode_alias`` @@ -152,6 +154,14 @@ def substitute_quoted( Only *quoted* occurrences move. A bare occurrence of the same text is a different identifier — a table alias, say — and is left alone, which is what keeps the deferred join-path-alias surface (DEV-1743) out of scope here. + + This is a string pass, not an AST pass, so it is not literal-aware. Being + keyed to an exact alias set rather than a length regex bounds the exposure + to one contrived case: a string literal containing the exact dialect-quoted + spelling of an over-limit alias *of the same query* (``note = '"Root.a.b. + <62 more bytes>"'``) would have its contents rewritten. A regex over quoted + spans — which the BigQuery/T-SQL dot-manglers already run over this same + SQL — has strictly wider exposure, so this pass does not add a risk class. """ if not mapping: return sql diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index cd79daa2..641627b8 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -618,7 +618,8 @@ def rewrite_emitted_sql( ``aliases`` whose fitted form differs has its dialect-quoted token replaced, everywhere it occurs (inner ``AS``, outer wrap projection, ORDER BY, CTE column references). Driven by the query's own alias set - rather than a length regex, so it cannot reach into a string literal. + rather than a length regex, which bounds what it can touch inside a + string literal — see :func:`substitute_quoted`. ``aliases`` defaults to empty, which makes this a no-op — every caller that does not supply an alias set keeps today's behaviour exactly. diff --git a/tests/test_dev1756_identifier_length.py b/tests/test_dev1756_identifier_length.py index 5dbf3c5c..fe689ea0 100644 --- a/tests/test_dev1756_identifier_length.py +++ b/tests/test_dev1756_identifier_length.py @@ -739,20 +739,72 @@ async def test_long_siblings_get_distinct_shorts(self, chain) -> None: names = [c.name for c in vm.columns] assert len(names) == len(set(names)) - async def test_short_aliases_are_quoted(self, chain) -> None: - """Emitted bare, a mixed-case short is case-folded by Postgres while - the outer stage references it quoted -> UndefinedColumnError. Quoting - the alias fixes that AND removes the case-fold collision exposure.""" + async def test_short_alias_quoting_matches_the_downstream_reference( + self, chain, + ) -> None: + """The wrapper's ``AS `` must be quoted exactly when the + downstream ``Column(sql=short)`` reference is. + + Emitted bare, a MIXED-CASE short is case-folded by Postgres while the + outer stage references it quoted (``_quote_mixed_case_identifiers``) + -> ``UndefinedColumnError``. But quoting unconditionally breaks the + mirror image on UPPER-folding backends: a case-sensitive ``"status"`` + would be defined while the bare reference resolves as ``STATUS``. The + contract is agreement, not quoting. + """ engine, _ = chain vm = await engine._query_as_model(inner_query=_repro_query()) + gen = SQLGenerator(dialect="postgres") select = _wrapper_select(vm.sql) assert select.expressions, vm.sql + by_short = {c.name: c for c in vm.columns} for proj in select.expressions: assert isinstance(proj, exp.Alias), f"{proj.sql()} is not an aliased projection" ident = proj.args.get("alias") assert isinstance(ident, exp.Identifier), f"{ident} is not an Identifier" - assert ident.quoted, ( - f"short alias {ident} must be dialect-quoted\n{vm.sql}" + short = ident.this + assert short in by_short, f"{short!r} is not a virtual-model column\n{vm.sql}" + # How the generator will emit a downstream reference to this column. + ref = gen._parse(by_short[short].sql).find(exp.Column) + assert ref is not None + assert ident.quoted == ref.this.quoted, ( + f"short {short!r} is emitted {'quoted' if ident.quoted else 'bare'} " + f"in the wrapper but referenced " + f"{'quoted' if ref.this.quoted else 'bare'} downstream — a " + f"case-folding backend resolves those to different columns\n{vm.sql}" + ) + + async def test_mixed_case_short_is_quoted(self, chain) -> None: + """The original DEV-1756 defect: a mixed-case short emitted bare.""" + engine, _ = chain + vm = await engine._query_as_model(inner_query=_repro_query()) + mixed = [ + proj for proj in _wrapper_select(vm.sql).expressions + if isinstance(proj, exp.Alias) + and any(c.isupper() for c in proj.args["alias"].this) + ] + assert mixed, f"fixture should produce mixed-case shorts\n{vm.sql}" + for proj in mixed: + assert proj.args["alias"].quoted, ( + f"mixed-case short {proj.args['alias'].this!r} must be quoted " + f"or Postgres folds it out from under the reference\n{vm.sql}" + ) + + async def test_lowercase_short_stays_bare(self, chain) -> None: + """The Snowflake/Oracle mirror image: an all-lowercase short must NOT + be quoted, or the case-sensitive definition stops matching the bare + reference that those backends fold to upper.""" + engine, _ = chain + vm = await engine._query_as_model(inner_query=_repro_query()) + lower = [ + proj for proj in _wrapper_select(vm.sql).expressions + if isinstance(proj, exp.Alias) + and not any(c.isupper() for c in proj.args["alias"].this) + ] + assert lower, f"fixture should produce lowercase shorts\n{vm.sql}" + for proj in lower: + assert not proj.args["alias"].quoted, ( + f"lowercase short {proj.args['alias'].this!r} must stay bare\n{vm.sql}" ) async def test_inner_and_wrapper_agree_on_the_fitted_alias(self, chain) -> None: @@ -816,6 +868,38 @@ async def test_two_dimensions_landing_on_one_short_raise(self, tmp_path) -> None assert "query-backed model column" in str(exc.value) assert flat in str(exc.value) + async def test_caller_supplied_measure_names_are_fitted(self, chain) -> None: + """A measure/transform/expression ``name`` is caller-supplied and + bypasses ``_alias_to_short``, so it needs its own fitting. + + These land in ``column_map`` verbatim, which means they reach the SQL as + ``AS ""`` AND become ``Column.name``. Two over-limit names sharing + a 63-byte prefix are truncated onto one column by Postgres, and the + ``short_owner`` check cannot see it — they differ as Python strings. + """ + engine, _ = chain + long_a = "z" * 63 + "b" # 64 bytes + long_b = "z" * 63 + "c" # 64 bytes, identical first 63 + assert long_a[:63] == long_b[:63] + query = SlayerQuery( + source_model="SandboxInvoiceV2", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "totalAmount:sum", "name": long_a}, + {"formula": "totalAmount:avg", "name": long_b}, + ], + ) + vm = await engine._query_as_model(inner_query=query) + names = [c.name for c in vm.columns] + for name in names: + assert _nbytes(name) <= 63, f"{name!r} is {_nbytes(name)} bytes" + # The real defect: distinct AFTER the server's 63-byte truncation. + truncated = [n.encode()[:63] for n in names] + assert len(set(truncated)) == len(truncated), ( + f"two shorts collapse onto one 63-byte name: {names}" + ) + assert long_a not in vm.sql and long_b not in vm.sql, vm.sql + async def test_nested_dag_two_levels_agree(self, chain) -> None: """Two stages: stage 2 references stage 1's virtual-model columns.""" engine, _ = chain diff --git a/tests/test_query_backed_models.py b/tests/test_query_backed_models.py index 0d988b43..15e64386 100644 --- a/tests/test_query_backed_models.py +++ b/tests/test_query_backed_models.py @@ -1418,13 +1418,14 @@ async def test_inner_stage_aggregated_measure_honors_user_name(self) -> None: f"expected 'rev_sum' in cached columns, got: {col_names}" ) sql = loaded.backing_query_sql or "" - # Inner-stage wrap renames `"orders.rev" AS "rev"`; loose match on - # the alias keyword + name (newline-tolerant). DEV-1756: the short - # is now ALWAYS dialect-quoted, so an unquoted mixed-case short is - # no longer case-folded out from under the outer stage's reference. + # Inner-stage wrap renames `"orders.rev" AS rev`; loose match on + # the alias keyword + name (newline-tolerant). DEV-1756: an + # all-lowercase short stays BARE so it folds the same way the + # downstream bare reference does; only mixed-case / reserved shorts + # are quoted. import re - assert re.search(r'\bAS\s+"rev"', sql), ( - f"expected inner-stage 'AS \"rev\"' rename in SQL:\n{sql}" + assert re.search(r"\bAS\s+rev\b", sql), ( + f"expected inner-stage 'AS rev' rename in SQL:\n{sql}" ) # The canonical name must not leak into the wrapped subquery's # exposed alias. @@ -1493,8 +1494,8 @@ async def test_query_as_model_emits_user_alias_unit(self) -> None: f"'name', got: {col_names}" ) import re - # DEV-1756: shorts are always dialect-quoted now. - assert re.search(r'\bAS\s+"rev"', virtual.sql), ( + # DEV-1756: an all-lowercase short stays bare (see above). + assert re.search(r"\bAS\s+rev\b", virtual.sql), ( f"wrapped SQL must rename to user alias 'rev':\n{virtual.sql}" ) finally: From f573c35072dc58ab04a4489c57ff481749a5b4f2 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 10:25:35 +0200 Subject: [PATCH 05/10] fix(DEV-1756): derive short-alias quoting from the dialect, not a hardcoded set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up review found the previous predicate (`short.lower() in SLAYER_RESERVED_KEYWORDS or any(c.isupper() ...)`) wrong on both of its axes for some dialects: * It misses words reserved only NATIVELY. `install_reserved_keywords` unions SLayer's set INTO each sqlglot generator's own, so only the generator knows the full set. On MySQL `index` / `int` / `rows` (and `rows` on BigQuery) are quoted by the downstream reference while the wrapper emitted them bare — and a bare `AS index` is a syntax error there, not merely a mismatch. * It ignores shape. `Column.name` forbids only `.` and `:`, so a short can be `1abc` or `foo bar`, which needs quoting on form alone regardless of case or reserved-ness. Rather than grow the predicate, `_short_sql` now runs the short through the same two mechanisms the downstream `Column(sql=short)` reference goes through: `SQLGenerator._maybe_quote_ident` (DEV-1645's uppercase rule) followed by `Identifier.sql(dialect=...)`, which lets sqlglot apply the target dialect's reserved-word and identifier-safety rules. The two spellings are then equal by construction rather than by a rule that has to be kept in sync. Pinned by a 12-dialect x 7-name test asserting the wrapper spelling and the reference spelling are byte-identical; it fails on 4 dialect/name pairs against the previous predicate. The remaining wrapper/reference disagreements are all names `_parse` cannot read as a column at all (`select`, `foo-bar`, `1abc`) — pre-existing, and unreferenceable however the wrapper spells them. Also splits a composite assertion flagged by Sonar (python:S9073). Co-Authored-By: Claude Opus 5 (1M context) --- DECISIONS.md | 2 +- slayer/engine/query_engine.py | 50 ++++++++++++++----------- tests/test_dev1756_identifier_length.py | 41 +++++++++++++++++++- 3 files changed, 70 insertions(+), 23 deletions(-) diff --git a/DECISIONS.md b/DECISIONS.md index bd70a504..b6189730 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -70,4 +70,4 @@ implementation detail. Include issue refs when known. - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. - 2026-08-05 — Ingestion sees views, survives unmodellable names, and stops being silent (DEV-1741). **Views**: `list_ingestable_objects` replaces the bare `get_table_names()` at every introspection site, adding `get_view_names` + `get_materialized_view_names` behind a `NotImplementedError`/`Exception` guard (the base `Inspector` RAISES for matviews on unsupporting dialects rather than returning `[]`), de-duplicated first-classification-wins because some dialects return views from `get_table_names()`, in a deterministic tables→views→matviews order that the name-collision policy depends on. Ingested by default — dbt materializes staging models as views, so opt-in would have left the reported failure in place for a fresh install — with `--no-views` on both `slayer ingest` and `datasources create --ingest`. The drift side (`_live_schema_for_datasource`) and the MCP listing (`_fetch_tables`) take **no flag and are unconditional**: that map is only ever a lookup target (`validate_datasource` iterates the *persisted* models, `available_in_ds` derives from them), so views there cannot manufacture a model or a drift entry — what they fix is a pre-existing **data-loss** bug where a hand-authored model whose `sql_table` named a view resolved to `live_table=None` → `WholeModelDelete` → deleted by `validate-models --force-clean`. Gating that on `--no-views` would re-arm it for exactly the users who opted out. **Names**: model names can't contain `__` (six modules — generator/enrichment/column_expansion/column_dependency/schema_drift/osi — `split("__")` an alias back into a join path, so a model named `a__b` is read as the alias for `a→b` and yields a silently wrong query, not a crash), but `sql_table` can, so a dlt child table `reports__patient__drug` is modelled as `reports_patient_drug` with `sql_table` verbatim. Sanitizer is `re.sub(r"_{2,}", "_")`, NOT `replace("__","_")` — `str.replace` is non-overlapping, so `a___b`→`a__b` would still fail validation. Collisions reserve every unsanitized name first (a real `a_b` always beats a sanitized `a__b`, order-independently) and **skip** rather than suffix, since suffixes shift as the object set changes and would orphan models and churn drift. A per-object `try/except` backstops everything else (`.`/`:`/`/`/`\` names, bad column names, per-object introspection failures); the FK-collection loop and `_get_fk_relationships` are guarded too because they run BEFORE that isolation and would otherwise still kill the run. **Reporting**: skips travel in a new `skipped` list, deliberately NOT folded into `errors` — `slayer ingest` exits 1 on either (we declined a perfectly valid object; `--exclude` is the documented remedy) but `POST /ingest` keeps 422 for `errors` only, because a permanent 422 aimed at a machine that can't act on the hint buries a successful partial ingest behind an error status. An empty scan prints the available schemas and exits 1 (the reporter listed the missing exit code as part of the defect), gated on `objects` not `additions` so a healthy no-op re-ingest stays quiet; `datasources create --ingest` on an empty DB still exits 0, since creating the datasource is that command's job and it succeeded. `in_scope_table_names` switched from model names to `_bare_table_name(sql_table)` — it is compared against table names in `_scoped_models_for_validation`, so the old keying silently dropped from validation scope any model whose name differs from its table (every sanitized model, and already every dbt/OSI hidden model passing `model_name=`). **`source_kind`** (`table`/`view`/`materialized_view`/`None`=unknown) persists on `SlayerModel`, v7→v8 with a no-op migration (mandatory: `migrate()` raises `RuntimeError` on an unregistered step); `None` for pre-v8, hand-authored and sql/query-backed models is the honest value, not a guess. It is a deliberate **exception to the additive-merge contract** — refreshed, not preserved — because it describes the live object rather than user intent, and the transition it exists to capture (dbt `+materialized: table`) usually changes no columns at all; the refresh therefore has to be made in three places (the early return, the `model_copy(update=...)`, and the save gate in `_process_one_table`), since editing only the update dict computes a corrected model and throws it away. A `None` from a non-classifying path never erases a known value. Docs-only fix for the advertised `motley-slayer[duckdb]` extra, which does not exist: `duckdb`/`duckdb-engine` are unconditional core deps (the Postgres facade imports duckdb at top level on every CLI invocation), and adding them under `[tool.poetry.extras]` would *gate* rather than alias them, breaking bare `pip install motley-slayer` → `datasources create demo`. -- 2026-08-06 — Dialect-aware identifier-length fitting (DEV-1756): every dialect declares a conservative universal budget as `SqlDialect.max_identifier_bytes` (postgres 63, mysql 64, redshift 127, oracle/tsql 128, snowflake 255, duckdb 256, bigquery 300, `None` = unbounded for sqlite/clickhouse/trino/presto/databricks/spark), and an over-limit identifier is shortened at emission to `__` via `slayer/sql/dialects/_identifier_fit.py`. Postgres is the binding case and the reason this is a correctness fix rather than cosmetics: it truncates over-length identifiers **silently** (a NOTICE, never an error), so two 3-hop aliases sharing a 63-byte prefix either blow up as `AmbiguousColumnError` under the DEV-1444 outer wrap or — with no sibling to collide with — quietly return the column under a name the engine never looks up. Shorten **only when over the limit**, never uniformly: `decode_result_keys` restores the canonical dotted alias, so the dialect-dependence the issue worried about is invisible to consumers and the 99% case keeps byte-identical SQL (pinned by pre-change goldens, not merely by idempotence). Both ends of the alias are kept because the reported colliding pair differs ONLY in its final segment — a head-only truncation would render the two indistinguishable in `dry_run` output. The digest is sha256 of the FULL original, which is what makes `fit_identifier` a pure function of the name and lets the read side rebuild the emitted→canonical map by re-running it, with no map threaded through generation (the alternative the issue sketched). Write side is an EXACT-match replacement over the query's own alias set (`all_projection_aliases`, unfiltered — hidden ORDER-BY hoists and `_inner_*`/`_ft*`/`_ts*` entries are projected in the inner SELECT and truncate identically), never a length regex over arbitrary SQL, so a long quoted-looking span inside a string literal can never be corrupted; substitution is two-phase (canonical→sentinel→final) as defence-in-depth, though today's key set (over-limit) and value set (within-limit) are provably disjoint. BigQuery/T-SQL compose by running their existing dot-mangle regex AFTER the base length pass: an under-limit alias makes the length pass a genuine no-op so their output is unchanged, and an over-limit one arrives still-dotted and gets mangled by the same regex — no double-encoding — with the budget sized against `encode_alias` so the post-mangle form still fits. Scope is the three OUTPUT-name surfaces: projection aliases, CTE names (`_cte_name_from_alias` fits the whole result, prefix included, allocated through a per-statement collision-checked `SQLGenerator._cte_name`), and `_query_as_model` virtual-model shorts. Join-path TABLE aliases are deferred to DEV-1743, whose plan already owns the `__` path-alias allocator; their failure mode (silent wrong joins) is worse but the fix requires decoupling `EnrichedDimension.model_name` from the emitted qualifier across ~8 sites that split it back on `__`. Collisions raise `IdentifierCollisionError` rather than emitting ambiguous SQL — the check covers identity entries too, since an already-short alias equal to another's fitted form is a duplicate no hash width can prevent. Fixed alongside, in the same code path: `_query_as_model` dialect-quotes its short alias when it is mixed-case as well as when it is a reserved word. Emitted bare, a mixed-case short was case-folded by Postgres while the outer stage referenced it quoted (DEV-1645 quotes mixed-case `Column.sql` leaves), making any query-backed model with a mixed-case join path unqueryable (`UndefinedColumnError`, reproduced on a live server). The contract is AGREEMENT between the two sides, not quoting: the policy deliberately mirrors `_quote_mixed_case_identifiers` + `prequote_reserved_identifiers`, which is what the downstream `Column(sql=short)` reference is emitted under. Quoting UNCONDITIONALLY was tried first and is wrong in the mirror direction — it defines a case-sensitive `"status"` on upper-folding backends (Snowflake, Oracle) while the bare reference still resolves as `STATUS`, so it trades a Postgres bug for a Snowflake one. Because an all-lowercase short therefore stays bare and still shares a case-folded namespace, the short-uniqueness check remains case-insensitive. +- 2026-08-06 — Dialect-aware identifier-length fitting (DEV-1756): every dialect declares a conservative universal budget as `SqlDialect.max_identifier_bytes` (postgres 63, mysql 64, redshift 127, oracle/tsql 128, snowflake 255, duckdb 256, bigquery 300, `None` = unbounded for sqlite/clickhouse/trino/presto/databricks/spark), and an over-limit identifier is shortened at emission to `__` via `slayer/sql/dialects/_identifier_fit.py`. Postgres is the binding case and the reason this is a correctness fix rather than cosmetics: it truncates over-length identifiers **silently** (a NOTICE, never an error), so two 3-hop aliases sharing a 63-byte prefix either blow up as `AmbiguousColumnError` under the DEV-1444 outer wrap or — with no sibling to collide with — quietly return the column under a name the engine never looks up. Shorten **only when over the limit**, never uniformly: `decode_result_keys` restores the canonical dotted alias, so the dialect-dependence the issue worried about is invisible to consumers and the 99% case keeps byte-identical SQL (pinned by pre-change goldens, not merely by idempotence). Both ends of the alias are kept because the reported colliding pair differs ONLY in its final segment — a head-only truncation would render the two indistinguishable in `dry_run` output. The digest is sha256 of the FULL original, which is what makes `fit_identifier` a pure function of the name and lets the read side rebuild the emitted→canonical map by re-running it, with no map threaded through generation (the alternative the issue sketched). Write side is an EXACT-match replacement over the query's own alias set (`all_projection_aliases`, unfiltered — hidden ORDER-BY hoists and `_inner_*`/`_ft*`/`_ts*` entries are projected in the inner SELECT and truncate identically), never a length regex over arbitrary SQL, so a long quoted-looking span inside a string literal can never be corrupted; substitution is two-phase (canonical→sentinel→final) as defence-in-depth, though today's key set (over-limit) and value set (within-limit) are provably disjoint. BigQuery/T-SQL compose by running their existing dot-mangle regex AFTER the base length pass: an under-limit alias makes the length pass a genuine no-op so their output is unchanged, and an over-limit one arrives still-dotted and gets mangled by the same regex — no double-encoding — with the budget sized against `encode_alias` so the post-mangle form still fits. Scope is the three OUTPUT-name surfaces: projection aliases, CTE names (`_cte_name_from_alias` fits the whole result, prefix included, allocated through a per-statement collision-checked `SQLGenerator._cte_name`), and `_query_as_model` virtual-model shorts. Join-path TABLE aliases are deferred to DEV-1743, whose plan already owns the `__` path-alias allocator; their failure mode (silent wrong joins) is worse but the fix requires decoupling `EnrichedDimension.model_name` from the emitted qualifier across ~8 sites that split it back on `__`. Collisions raise `IdentifierCollisionError` rather than emitting ambiguous SQL — the check covers identity entries too, since an already-short alias equal to another's fitted form is a duplicate no hash width can prevent. Fixed alongside, in the same code path: `_query_as_model` no longer decides its short alias's quoting with a hand-written predicate. The contract is AGREEMENT between the emitted `AS ` and the downstream `Column(sql=short)` reference — they must resolve to the same column — so the short is now run through the same two mechanisms the reference side uses: `SQLGenerator._maybe_quote_ident` (DEV-1645: quote iff the name contains an uppercase letter) followed by `Identifier.sql(dialect=...)`, which lets sqlglot add quotes for words reserved IN THE TARGET DIALECT and for names that are not safe bare identifiers. Emitted bare, a mixed-case short was case-folded by Postgres while the outer stage referenced it quoted, making any query-backed model with a mixed-case join path unqueryable (`UndefinedColumnError`, reproduced on a live server). Two wrong answers were tried and rejected on the way: quoting UNCONDITIONALLY defines a case-sensitive `"status"` on upper-folding backends (Snowflake, Oracle) while the bare reference still resolves as `STATUS`, trading a Postgres bug for a Snowflake one; and quoting on `SLAYER_RESERVED_KEYWORDS` alone misses words reserved only NATIVELY (`index`, `int`, `rows` on MySQL; `rows` on BigQuery), where the reference side quotes but a bare `AS index` is a syntax error — `install_reserved_keywords` unions our set INTO each generator's, so only the generator knows the full set. Shape matters too: `Column.name` forbids only `.` and `:`, so `1abc` or `foo bar` needs quoting on form alone. Both axes must therefore be dialect-driven, and the rule is pinned by a 12-dialect × 7-name test asserting the two spellings are byte-identical. Because an all-lowercase short stays bare and still shares a case-folded namespace, the short-uniqueness check remains case-insensitive. diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index b5ac5074..bd08b5ac 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -81,7 +81,6 @@ from slayer.sql import engine_factory from slayer.sql.engine_factory import _runtime_fingerprint from slayer.sql.generator import SQLGenerator -from slayer.sql.reserved_keywords import SLAYER_RESERVED_KEYWORDS from slayer.sql.session_policy import ScopedTable, apply_session_policy from slayer.storage.base import StorageBackend @@ -3321,28 +3320,37 @@ def _fit_short(name: str) -> str: # their dotted aliases mangled. Hardcoded ANSI double quotes # would either fail to parse (MySQL) or reference an alias the # mangled inner subquery doesn't expose (T-SQL). - # DEV-1686: the inner ``alias`` is always dialect-quoted; the ``short`` - # output alias is quoted under exactly the policy the DOWNSTREAM - # reference uses, because the two have to agree. + # DEV-1686: the inner ``alias`` is always dialect-quoted. The ``short`` + # output alias instead has to be spelled EXACTLY as the downstream + # reference to it will be, because the two must resolve to the same + # column. So rather than approximate that policy, run the short through + # the same two mechanisms the reference side uses: # - # A downstream ``Column(sql=short)`` is parsed by the generator, where - # ``_quote_mixed_case_identifiers`` (DEV-1645) quotes a leaf iff it - # contains an uppercase letter, and ``prequote_reserved_identifiers`` - # (DEV-1686) quotes reserved words. So: - # * reserved / mixed-case -> quoted BOTH sides. This is the DEV-1756 - # fix: emitted bare, a mixed-case short was case-folded by Postgres - # while the outer stage referenced it quoted, making any - # query-backed model with a mixed-case join path or column - # unqueryable (``UndefinedColumnError``, seen on a live server). - # * all-lowercase -> bare BOTH sides, so it folds consistently. - # Quoting unconditionally instead breaks the lowercase case on - # UPPER-folding backends (Snowflake, Oracle): the wrapper would define a - # case-sensitive ``"status"`` while the bare reference resolves as - # ``STATUS``. + # * ``_maybe_quote_ident`` — DEV-1645's rule, quote iff the name + # contains an uppercase letter. This is the DEV-1756 fix: emitted + # bare, a mixed-case short was case-folded by Postgres while the + # outer stage referenced it quoted, making any query-backed model + # with a mixed-case join path or column unqueryable + # (``UndefinedColumnError``, seen on a live server). + # * ``Identifier.sql(dialect=...)`` — sqlglot adds quotes for words + # reserved IN THE TARGET DIALECT (``install_reserved_keywords`` + # unions SLayer's set into each generator's native one) and for + # names that are not safe bare identifiers. + # + # Both axes must be dialect-driven, not hardcoded. Checking only + # ``SLAYER_RESERVED_KEYWORDS`` misses MySQL-native words like ``index`` + # / ``int`` / ``rows``, which the reference side quotes and a bare + # ``AS index`` makes a syntax error; and a name like ``1abc`` or + # ``foo bar`` (``Column.name`` only forbids ``.`` and ``:``) needs + # quoting on shape alone. + # + # Quoting UNCONDITIONALLY is wrong in the other direction: it defines a + # case-sensitive ``"status"`` while the bare reference still resolves as + # ``STATUS`` on upper-folding backends (Snowflake, Oracle). def _short_sql(short: str) -> str: - if short.lower() in SLAYER_RESERVED_KEYWORDS or any(c.isupper() for c in short): - return exp.Identifier(this=short, quoted=True).sql(dialect=dialect) - return short + ident = exp.Identifier(this=short, quoted=False) + SQLGenerator._maybe_quote_ident(ident) + return ident.sql(dialect=dialect) # DEV-1756: the shorts share one output-column namespace, and each also # becomes a ``Column.name`` on the virtual model. Validate the whole diff --git a/tests/test_dev1756_identifier_length.py b/tests/test_dev1756_identifier_length.py index fe689ea0..92b8c6c6 100644 --- a/tests/test_dev1756_identifier_length.py +++ b/tests/test_dev1756_identifier_length.py @@ -790,6 +790,44 @@ async def test_mixed_case_short_is_quoted(self, chain) -> None: f"or Postgres folds it out from under the reference\n{vm.sql}" ) + @pytest.mark.parametrize("dialect", [ + "postgres", "mysql", "snowflake", "tsql", "bigquery", "duckdb", + "sqlite", "clickhouse", "redshift", "oracle", "trino", "spark", + ]) + @pytest.mark.parametrize("short", [ + "status", # plain lowercase -> bare on both sides + "SandboxConsumer__name", # mixed case -> quoted on both sides + "order", # reserved in SLayer's common set + "index", # reserved NATIVELY in mysql/tsql, not in ours + "int", # ditto + "rows", # ditto (also bigquery) + "_fit_a1b2c3d4_tail", # the shape fit_identifier emits + ]) + def test_short_spelling_matches_the_reference_on_every_dialect( + self, dialect: str, short: str, + ) -> None: + """``AS `` must be spelled exactly as a downstream + ``Column(sql=short)`` reference to it will be, on every dialect. + + Two independent axes, both dialect-driven: + * CASE — a bare mixed-case short is folded by the server while the + reference is quoted (the DEV-1756 defect), and quoting a lowercase + one breaks the mirror image on upper-folding backends. + * RESERVED / UNSAFE — ``index``/``int``/``rows`` are reserved in + MySQL but not in ``SLAYER_RESERVED_KEYWORDS``; the reference side + quotes them and a bare ``AS index`` is a syntax error. Checking + SLayer's set alone is not enough. + """ + gen = SQLGenerator(dialect=dialect) + ident = exp.Identifier(this=short, quoted=False) + SQLGenerator._maybe_quote_ident(ident) + wrapper = ident.sql(dialect=gen.dialect) + reference = gen._parse(short).sql(dialect=gen.dialect) + assert wrapper == reference, ( + f"[{dialect}] wrapper emits {wrapper!r} but a downstream reference " + f"to the same column emits {reference!r}" + ) + async def test_lowercase_short_stays_bare(self, chain) -> None: """The Snowflake/Oracle mirror image: an all-lowercase short must NOT be quoted, or the case-sensitive definition stops matching the bare @@ -898,7 +936,8 @@ async def test_caller_supplied_measure_names_are_fitted(self, chain) -> None: assert len(set(truncated)) == len(truncated), ( f"two shorts collapse onto one 63-byte name: {names}" ) - assert long_a not in vm.sql and long_b not in vm.sql, vm.sql + assert long_a not in vm.sql, vm.sql + assert long_b not in vm.sql, vm.sql async def test_nested_dag_two_levels_agree(self, chain) -> None: """Two stages: stage 2 references stage 1's virtual-model columns.""" From 2d887d5735060177b52d7720e018d1b0843e4ddb Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 11:02:45 +0200 Subject: [PATCH 06/10] fix(DEV-1756): close three identifier-fitting gaps found in review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit round on f573c350. Three real gaps, each with a test that fails without its fix: * `get_column_types` probed with generated SQL — which now carries FITTED aliases — but looked the results up by the canonical `EnrichedMeasure.alias`. An over-limit measure silently vanished from the type map (verified: the map came back empty). Decoded with the same hook and alias set `_run_and_build` uses. * `shifted_` / `sjoin_` self-join CTE names were f-string-built from a USER-supplied transform name, bypassing both the length fitting and the collision check. A 78-char transform name produced 86-byte CTE identifiers on Postgres while the projection alias beside them was correctly fitted. Routed through `_cte_name()`, so definition and every reference share one allocation. * The `_cte_name` allocator keyed on the literal emitted name, but CTE names are emitted UNQUOTED — so `_wm_Foo` and `_wm_foo` are one identifier on Postgres and both were accepted, silently pointing a reference at the wrong CTE. Keyed casefolded. Unlike the projection-alias namespace there is no quoted variant to exempt, so here casefolding is exact rather than merely conservative. Also: * `test_cte_allocator_resets_per_statement` asserted nothing and cleared `_cte_names` by hand. It now seeds an owner the statement does not use and drives the reset through `generate()` — regenerating the same statement proves nothing, since `_cte_name` is idempotent for one owner. * Corrected "every engine caps identifier length" in docs/concepts/queries.md; four dialects are unbounded, as the table two files over already says. * Keyword arguments at the multi-parameter call sites this PR added, per the repo convention. Not taken: storing `_short_sql(short)` in `Column.sql`. Unsafe-shaped shorts are unreachable — `ColumnRef.name`, `ModelMeasure.name` and `Aggregation.name` all enforce the full identifier pattern, so a query can neither reference nor name one. Pre-quoting `Column.sql` would instead double-quote it relative to the downstream reference and desync the pair f573c350 just aligned. Rationale posted on the thread. Co-Authored-By: Claude Opus 5 (1M context) --- docs/concepts/queries.md | 2 +- slayer/engine/query_engine.py | 22 ++++ slayer/sql/dialects/base.py | 6 +- slayer/sql/dialects/bigquery.py | 6 +- slayer/sql/dialects/tsql.py | 6 +- slayer/sql/generator.py | 33 ++++-- tests/test_dev1756_identifier_length.py | 128 +++++++++++++++++++++++- 7 files changed, 177 insertions(+), 26 deletions(-) diff --git a/docs/concepts/queries.md b/docs/concepts/queries.md index 1cd6a4b6..c6b5e1e7 100644 --- a/docs/concepts/queries.md +++ b/docs/concepts/queries.md @@ -133,7 +133,7 @@ Query results are returned as a `SlayerResponse`: Result keys are always this canonical dotted form, on every backend. Where a database's own rules force a different spelling in the emitted SQL — BigQuery -and SQL Server reject dotted column aliases, and every engine caps identifier +and SQL Server reject dotted column aliases, and most engines cap identifier length — the alias is rewritten on the way out and restored on the way back, so `data` and `columns` do not vary by dialect. Only `sql` shows the rewritten form, since that is what actually ran. See diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index bd08b5ac..dc8716ec 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -2053,6 +2053,15 @@ async def get_column_types( logger.warning("get_column_types probe failed for model '%s'", model_name) return {} + # DEV-1756: ``sql`` was generated with length-fitted aliases, so the + # probe's metadata keys are the EMITTED names while ``em.alias`` below + # is canonical. Decode first or an over-limit measure silently drops + # out of the type map. Same hook and alias set ``_run_and_build`` uses. + if raw_types: + raw_types = get_dialect(dialect).decode_result_keys( + [raw_types], aliases=all_projection_aliases(enriched), + )[0] + # Map qualified aliases (e.g., "orders.revenue_max") back to bare measure names result: dict[str, str] = {} for em in enriched.measures: @@ -3360,6 +3369,19 @@ def _short_sql(short: str) -> str: # prevent — two distinct aliases landing on the identical short because # a user-declared cross-model ``name`` bypassed the flattening above # and happened to match another column's canonical flat. + # + # Keyed on the CASEFOLDED short, which is deliberately CONSERVATIVE + # rather than exact. Since ``_short_sql`` quotes mixed-case shorts, + # ``Foo`` (emitted ``"Foo"``, case-preserved) and ``foo`` (emitted bare, + # folded) do in fact resolve to different columns on every dialect, so + # casefolding rejects a pair that would work. The exact rule is + # dialect-dependent — a quoted ``"EMAIL"`` collides with a bare + # ``email`` on upper-folding backends (Snowflake, Oracle) but not on + # lower-folding ones — and modelling that needs a fold-direction per + # dialect, which SLayer does not carry. Erring toward a loud error on a + # rare case-differing pair is the right side to be wrong on here: the + # alternative normalisation admits a silent duplicate output name on + # exactly the backends this machinery exists to protect. short_owner: dict[str, str] = {} for alias, short, _, _, _, _ in column_map: prior_alias = short_owner.setdefault(short.casefold(), alias) diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 641627b8..ca29fe02 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -524,7 +524,7 @@ def fit_alias(self, name: str) -> str: under-limit alias produces byte-identical SQL on every dialect, including the ones that separately mangle dots. """ - return fit_identifier(name, limit=self.max_identifier_bytes) + return fit_identifier(name=name, limit=self.max_identifier_bytes) def emit_alias(self, alias: str) -> str: """The FINAL identifier a canonical alias reaches the SQL as. @@ -634,7 +634,7 @@ def rewrite_emitted_sql( mapping = self.alias_rewrite_map(aliases) if not mapping: return sql - return substitute_quoted(sql, mapping, quote=self.quote_identifier) + return substitute_quoted(sql=sql, mapping=mapping, quote=self.quote_identifier) def decode_result_keys( self, @@ -655,7 +655,7 @@ def decode_result_keys( mapping = self.decode_alias_map(aliases) if not mapping: return rows - return [self._rekey_row(row, mapping) for row in rows] + return [self._rekey_row(row=row, mapping=mapping) for row in rows] def register_udfs(self, dbapi_connection) -> None: """Default: no-op. SQLite overrides to register Python aggregate diff --git a/slayer/sql/dialects/bigquery.py b/slayer/sql/dialects/bigquery.py index 0373f13c..d640125a 100644 --- a/slayer/sql/dialects/bigquery.py +++ b/slayer/sql/dialects/bigquery.py @@ -142,7 +142,7 @@ def fit_alias(self, name: str) -> str: the mangling. """ return fit_identifier( - name, limit=self.max_identifier_bytes, expand=encode_alias, + name=name, limit=self.max_identifier_bytes, expand=encode_alias, ) def emit_alias(self, alias: str) -> str: @@ -169,7 +169,7 @@ def rewrite_emitted_sql( Because the fitted form is mangled by this same pass rather than arriving pre-mangled, there is no double-encoding. """ - sql = super().rewrite_emitted_sql(sql, aliases=aliases) + sql = super().rewrite_emitted_sql(sql=sql, aliases=aliases) return _DOTTED_ALIAS_RE.sub( lambda m: f"`{encode_alias(m.group(1))}`", sql ) @@ -193,7 +193,7 @@ def decode_result_keys( """ mapping = self.decode_alias_map(aliases) return [ - self._rekey_row(row, mapping, fallback=decode_alias) + self._rekey_row(row=row, mapping=mapping, fallback=decode_alias) for row in rows ] diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index ac05cd64..29d15b85 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -345,7 +345,7 @@ def fit_alias(self, name: str) -> str: it on a deep chain. The returned value is still dotted. """ return fit_identifier( - name, limit=self.max_identifier_bytes, expand=encode_alias, + name=name, limit=self.max_identifier_bytes, expand=encode_alias, ) def emit_alias(self, alias: str) -> str: @@ -376,7 +376,7 @@ def rewrite_emitted_sql( still dotted, so this pass mangles it — yielding what ``emit_alias`` returns, with no double-encoding. """ - sql = super().rewrite_emitted_sql(sql, aliases=aliases) + sql = super().rewrite_emitted_sql(sql=sql, aliases=aliases) return _TSQL_DOTTED_ALIAS_RE.sub( lambda m: f"[{encode_alias(m.group(1))}]", sql ) @@ -400,6 +400,6 @@ def decode_result_keys( """ mapping = self.decode_alias_map(aliases) return [ - self._rekey_row(row, mapping, fallback=decode_alias) + self._rekey_row(row=row, mapping=mapping, fallback=decode_alias) for row in rows ] diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index a0a6a135..bf7a6c56 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -284,7 +284,7 @@ def _cte_name_from_alias(prefix: str, alias: str, *, limit: int | None = None) - """ sanitized = alias.replace(".", "__") sanitized = re.sub(r"[^a-zA-Z0-9_]", "_", sanitized) - return fit_identifier(prefix + sanitized, limit=limit) + return fit_identifier(name=prefix + sanitized, limit=limit) def _alias_prefixes(model_name: str) -> list: @@ -430,12 +430,19 @@ def _cte_name(self, prefix: str, alias: str) -> str: silently references the wrong CTE. Repeat calls with the same ``(prefix, alias)`` are the same name, not a collision — several code paths re-derive a CTE's name to reference it. + + Keyed CASEFOLDED, because unquoted is exactly the case that the server + folds: ``_wm_Foo`` and ``_wm_foo`` are one identifier on Postgres (and + on Snowflake), so comparing the literal spellings would wave through a + pair that silently resolves to the same CTE. Unlike the projection + aliases, there is no quoted variant to exempt here — these are never + quoted — so casefolding is exact rather than merely conservative. """ name = _cte_name_from_alias( - prefix, alias, limit=self._dialect.max_identifier_bytes, + prefix=prefix, alias=alias, limit=self._dialect.max_identifier_bytes, ) owner = (prefix, alias) - prior = self._cte_names.setdefault(name, owner) + prior = self._cte_names.setdefault(name.casefold(), owner) if prior != owner: raise IdentifierCollisionError( first=f"{prior[0]}{prior[1]}", second=f"{prefix}{alias}", @@ -802,7 +809,7 @@ def _build_combined(self, enriched: EnrichedQuery, # --- Cross-model measure CTEs --- seen_cm_ctes: set = set() for cm in enriched.cross_model_measures: - cte_name = self._cte_name("_cm_", cm.alias) + cte_name = self._cte_name(prefix="_cm_", alias=cm.alias) if cte_name in seen_cm_ctes: measure_cte_refs.append((cte_name, cm.alias, None)) continue @@ -887,7 +894,7 @@ def _build_combined(self, enriched: EnrichedQuery, for measure in enriched.measures: if not _is_windowed_measure(measure): continue - cte_name = self._cte_name("_wm_", measure.alias) + cte_name = self._cte_name(prefix="_wm_", alias=measure.alias) ctes.append((cte_name, self._generate_window_measure_cte(enriched=enriched, measure=measure))) measure_cte_refs.append((cte_name, measure.alias, None)) @@ -895,7 +902,7 @@ def _build_combined(self, enriched: EnrichedQuery, for measure in enriched.measures: if not _has_cross_model_filter(measure): continue - cte_name = self._cte_name("_fm_", measure.alias) + cte_name = self._cte_name(prefix="_fm_", alias=measure.alias) # Measure aggregation without CASE WHEN (the join IS the filter) unfiltered = copy.copy(measure) @@ -1681,7 +1688,11 @@ def _generate_with_computed(self, enriched: EnrichedQuery, for t in deferred_self_joins: src_cte = ctes[-1][0] - shift_name = f"shifted_{t.name}" + # DEV-1756: these are CTE names built from a USER-supplied + # transform name, so they need the same length fitting and + # collision check as every other CTE. Allocated once and reused + # for the definition and every reference below. + shift_name = self._cte_name(prefix="shifted_", alias=t.name) shifted_sql = self._generate_shifted_base( enriched=enriched, transform=t, ) @@ -1703,7 +1714,7 @@ def _generate_with_computed(self, enriched: EnrichedQuery, join_cols = ", ".join( f'{src_cte}.{self._q(a)}' for a in sorted(available_aliases) ) - join_layer = f"sjoin_{t.name}" + join_layer = self._cte_name(prefix="sjoin_", alias=t.name) join_sql = ( f"SELECT {join_cols}, {col_sql} AS {self._q(t.alias)}\n" f"FROM {src_cte}\n" @@ -1821,9 +1832,9 @@ def _build_consecutive_periods_ctes( layer_num: int, ) -> tuple[list[tuple[str, str]], list[tuple[str, str]]]: partition_aliases = getattr(transform, "partition_aliases", []) or [] - reset_alias = self._cte_name("_cp_reset_", transform.alias) - reset_cte = self._cte_name(f"cp_reset_{layer_num}_", transform.alias) - value_cte = self._cte_name(f"cp_value_{layer_num}_", transform.alias) + reset_alias = self._cte_name(prefix="_cp_reset_", alias=transform.alias) + reset_cte = self._cte_name(prefix=f"cp_reset_{layer_num}_", alias=transform.alias) + value_cte = self._cte_name(prefix=f"cp_value_{layer_num}_", alias=transform.alias) def _quoted_col(name: str) -> exp.Column: return exp.Column(this=exp.to_identifier(name, quoted=True)) diff --git a/tests/test_dev1756_identifier_length.py b/tests/test_dev1756_identifier_length.py index 92b8c6c6..124232f6 100644 --- a/tests/test_dev1756_identifier_length.py +++ b/tests/test_dev1756_identifier_length.py @@ -658,13 +658,41 @@ def test_cte_allocator_is_idempotent_for_the_same_owner(self) -> None: second = gen._cte_name("_cm_", TWIN_A) # hits the memo, must not raise assert second == first - def test_cte_allocator_resets_per_statement(self) -> None: + async def test_cte_allocator_resets_per_statement(self, chain) -> None: """One generator instance generates many statements; allocation is - per-statement, so names must not accumulate across calls.""" + per-statement, so an owner allocated before ``generate()`` must not + still hold its name afterwards. + + The reset has to be exercised THROUGH ``generate()``, and with a + DIFFERENT owner: re-generating the same statement proves nothing, + because ``_cte_name`` is idempotent for one owner and would pass even + if the dict were never cleared. + """ + engine, _ = chain + prepared = await engine._prepare_pipeline( + query=_repro_query(), named_queries={}, runtime_kwarg={}, + ) gen = SQLGenerator(dialect="postgres") - gen._cte_name("_cm_", TWIN_A) - gen._cte_names = {} - gen._cte_name("_cm_", TWIN_A) # would raise if state leaked wrongly + stale = gen._cte_name(prefix="_cm_", alias=TWIN_A) + assert stale.casefold() in gen._cte_names + + gen.generate(enriched=prepared.enriched) + assert stale.casefold() not in gen._cte_names, ( + "CTE allocation leaked across statements" + ) + + def test_cte_allocator_detects_case_folded_collision(self) -> None: + """CTE names are emitted UNQUOTED, so the server folds them: on + Postgres ``_wm_Foo`` and ``_wm_foo`` are the SAME identifier, and + allocating both would silently point one reference at the other's CTE. + Comparing the literal spellings misses it. + """ + gen = SQLGenerator(dialect="postgres") + first = gen._cte_name(prefix="_wm_", alias="Foo") + with pytest.raises(IdentifierCollisionError) as exc: + gen._cte_name(prefix="_wm_", alias="foo") + assert "CTE name" in str(exc.value) + assert first == "_wm_Foo" def test_cte_name_helper_is_pure_and_bounded(self) -> None: from slayer.sql.generator import _cte_name_from_alias @@ -710,6 +738,47 @@ def test_cte_name_is_a_legal_unquoted_identifier(self) -> None: got = _cte_name_from_alias("_cm_", LONG_EMAIL + ".lifetimeValue_sum", limit=63) assert re.match(r"^[A-Za-z_][A-Za-z0-9_]*$", got), got + async def test_self_join_transform_cte_names_are_fitted(self, tmp_path) -> None: + """``shifted_`` / ``sjoin_`` are CTE names built from a + USER-supplied transform name, so a long one busts the budget just like + an alias-derived CTE. They used to be f-string-built and so escaped + both the fitting and the collision check. + """ + long_transform_name = "revenue_" + "x" * 70 # 78 chars, way over 63 + storage = YAMLStorage(base_dir=str(tmp_path)) + await storage.save_datasource(DatasourceConfig( + name=DS, type="postgres", host="localhost", port=5432, + database="x", username="u", password="p", + )) + await storage.save_model(SlayerModel( + name="ShiftOrders", sql_table="orders", data_source=DS, + default_time_dimension="created_at", + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="created_at", sql="created_at", type=DataType.TIMESTAMP), + Column(name="revenue", sql="revenue", type=DataType.DOUBLE), + ], + )) + engine = SlayerQueryEngine(storage=storage) + prepared = await engine._prepare_pipeline( + query=SlayerQuery( + source_model="ShiftOrders", + time_dimensions=[{"dimension": {"name": "created_at"}, "granularity": "month"}], + measures=[ + {"formula": "revenue:sum"}, + {"formula": "time_shift(revenue:sum, -1)", "name": long_transform_name}, + ], + ), + named_queries={}, runtime_kwarg={}, + ) + sql = prepared.sql + assert "shifted_" in sql, f"fixture must emit a self-join CTE\n{sql}" + for name, _ in _cte_names(sqlglot.parse_one(sql, dialect="postgres")): + assert _nbytes(name) <= 63, f"CTE {name!r} is {_nbytes(name)} bytes\n{sql}" + # The unfitted forms must not survive anywhere — definition or reference. + assert f"shifted_{long_transform_name}" not in sql + assert f"sjoin_{long_transform_name}" not in sql + # =========================================================================== # 6. Surface 4 — _query_as_model short names @@ -980,6 +1049,55 @@ async def test_attribute_keys_are_result_keys(self, chain) -> None: for key in resp.attributes.dimensions: assert key in resp.columns + async def test_get_column_types_decodes_fitted_aliases( + self, chain, monkeypatch: pytest.MonkeyPatch, + ) -> None: + """``get_column_types`` probes with generated SQL, so its metadata keys + are the EMITTED (fitted) aliases while the lookup below uses the + canonical ``EnrichedMeasure.alias``. Without a decode pass an + over-limit measure silently drops out of the type map. + """ + engine, _ = chain + over_limit = "totalAmount_" + "x" * 60 # 72-char measure name + + captured: dict[str, str] = {} + + class _FakeClient: + async def get_column_types(self, sql: str) -> dict[str, str]: + # Echo back what a server would: keys exactly as emitted. + for name, _ in _projection_aliases( + next(iter(sqlglot.parse_one(sql, dialect="postgres").find_all(exp.Select))) + ): + captured[name] = "double precision" + return dict(captured) + + async def aclose(self) -> None: # pragma: no cover — not reached + pass + + model = SlayerModel( + name="TypeProbe", sql_table="invoices", data_source=DS, + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name=over_limit, sql="total_amount", type=DataType.DOUBLE), + ], + ) + await engine.storage.save_model(model) + monkeypatch.setattr( + engine, "_sql_clients", {k: _FakeClient() for k in ("x",)}, + ) + monkeypatch.setattr( + "slayer.engine.query_engine._sql_client_cache_key", lambda ds: "x", + ) + + types = await engine.get_column_types(model_name="TypeProbe") + assert captured, "probe must have emitted at least one alias" + emitted = [k for k in captured if _nbytes(k) > 63] + assert not emitted, f"probe SQL should carry fitted aliases, got {emitted}" + assert over_limit in types, ( + f"over-limit measure dropped from the type map; probe keys were " + f"{sorted(captured)}" + ) + # =========================================================================== # 8. Sweep — every generator shape, not just the reported one From 1f7c237586e45240141f129e73371fe06a04cf8f Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 11:49:52 +0200 Subject: [PATCH 07/10] refactor(DEV-1756): drop the redundant guard tipping get_column_types over S3776 The decode added in 2d887d57 came with an `if raw_types:` guard, which pushed the function's cognitive complexity from 15 to 16 (Sonar python:S3776). The guard was never needed: `decode_result_keys` returns its input untouched when the alias map is empty, and re-keying an empty dict yields an empty dict. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/query_engine.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index dc8716ec..b1769a80 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -2057,10 +2057,11 @@ async def get_column_types( # probe's metadata keys are the EMITTED names while ``em.alias`` below # is canonical. Decode first or an over-limit measure silently drops # out of the type map. Same hook and alias set ``_run_and_build`` uses. - if raw_types: - raw_types = get_dialect(dialect).decode_result_keys( - [raw_types], aliases=all_projection_aliases(enriched), - )[0] + # Unconditional — the hook returns its input untouched when nothing was + # shortened, so an empty ``raw_types`` needs no guard. + raw_types = get_dialect(dialect).decode_result_keys( + [raw_types], aliases=all_projection_aliases(enriched), + )[0] # Map qualified aliases (e.g., "orders.revenue_max") back to bare measure names result: dict[str, str] = {} From 634f7821b3747c23d088d70a87dfd10afa4d7577 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 13:12:48 +0200 Subject: [PATCH 08/10] style(DEV-1756): keyword-arg the last two rewrite_emitted_sql call sites Both remaining multi-parameter calls this PR added were still passing `sql` positionally. Completes the convention sweep from 2d887d57. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/query_engine.py | 2 +- slayer/sql/generator.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index b1769a80..2e32d9d6 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -3408,7 +3408,7 @@ def _short_sql(short: str) -> str: # token inside ``inner_sql``, so this pass can only reach the wrapper's # own references. wrapped_sql = get_dialect(dialect).rewrite_emitted_sql( - wrapped_sql, aliases=all_projection_aliases(enriched), + sql=wrapped_sql, aliases=all_projection_aliases(enriched), ) # One Column per result column — each is potentially both a dimension diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index bf7a6c56..fc7f5a67 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -686,7 +686,7 @@ def generate( # SELECT and truncate exactly like user-declared aliases, so a filtered # list would leave their references pointing at an unfitted name. sql = self._dialect.rewrite_emitted_sql( - sql, aliases=all_projection_aliases(enriched), + sql=sql, aliases=all_projection_aliases(enriched), ) return sql From ef1b57c6f796c7bf8274df6bad479d787fb6e6d5 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Tue, 11 Aug 2026 16:48:19 +0200 Subject: [PATCH 09/10] docs(DEV-1756): cut over-verbose comments and docstrings (~3x) Trim the identifier-fitting comments/docstrings that read like design essays down to the essential why. Verified comment/docstring-only: code AST is byte-identical, full non-integration suite still green. Co-Authored-By: Claude Opus 4.8 --- slayer/core/errors.py | 14 +- slayer/engine/enriched.py | 14 +- slayer/engine/query_engine.py | 140 ++------- slayer/sql/dialects/_identifier_fit.py | 123 ++------ slayer/sql/dialects/_tier2.py | 16 +- slayer/sql/dialects/base.py | 100 ++----- slayer/sql/dialects/bigquery.py | 49 +-- slayer/sql/dialects/clickhouse.py | 3 +- slayer/sql/dialects/duckdb.py | 3 +- slayer/sql/dialects/mysql.py | 2 +- slayer/sql/dialects/postgres.py | 2 +- slayer/sql/dialects/snowflake.py | 1 - slayer/sql/dialects/sqlite.py | 3 +- slayer/sql/dialects/tsql.py | 50 +--- slayer/sql/generator.py | 60 ++-- tests/dialects/test_identifier_fit.py | 165 +++-------- .../test_dev1756_identifier_length_pg.py | 61 +--- tests/test_dev1756_identifier_length.py | 279 ++++-------------- tests/test_query_backed_models.py | 7 +- 19 files changed, 259 insertions(+), 833 deletions(-) diff --git a/slayer/core/errors.py b/slayer/core/errors.py index c38800ef..26f63fef 100644 --- a/slayer/core/errors.py +++ b/slayer/core/errors.py @@ -152,17 +152,11 @@ def __init__( class IdentifierCollisionError(SlayerError, ValueError): - """Raised when two distinct SLayer-generated names collapse onto one - identifier after the dialect's length fitting (DEV-1756). + """Two distinct SLayer-generated names collapse onto one identifier after + the dialect's length fitting (DEV-1756). - Fitting appends a 32-bit digest of the full original, so this is - astronomically unlikely — but a duplicate output name on a backend that - silently truncates is exactly the class of bug this machinery exists to - prevent, so it is raised loudly rather than left to corrupt a result set. - Also covers the case where an already-short name happens to equal another - name's fitted form, which hash width alone cannot prevent. - - Multi-inherits ``ValueError`` to match the other SLayer validation errors. + Raised loudly rather than left to corrupt a result set. Also covers an + already-short name equal to another name's fitted form. """ def __init__( diff --git a/slayer/engine/enriched.py b/slayer/engine/enriched.py index 9d72c8f5..16a1c3d7 100644 --- a/slayer/engine/enriched.py +++ b/slayer/engine/enriched.py @@ -277,16 +277,10 @@ class CrossModelMeasure(BaseModel): def all_projection_aliases(enriched: EnrichedQuery) -> list[str]: """Every alias ``enriched`` can put into an emitted SELECT, in bucket order. - DEV-1756: the superset of :func:`public_projection_aliases`, WITHOUT the - internal-prefix filtering. Hidden entries — ``_inner_*`` nested-transform - arg hoists, ``_ft*`` filter-transform extractions, ``_ts*`` change/change_pct - desugars, ORDER-BY aggregate hoists — are still projected in the inner - SELECT, so they truncate exactly like a user-declared alias and must be - length-fitted with the same map. Feeding the write pass a filtered list - would leave those references pointing at an unfitted name. - - Deduplicated (an alias can be reachable through more than one bucket) while - preserving first-seen order, so the derived rewrite map is deterministic. + Superset of :func:`public_projection_aliases` WITHOUT internal-prefix + filtering: hidden entries (``_inner_*``/``_ft*``/``_ts*`` hoists, ORDER-BY + aggregates) are still projected and must be length-fitted with the same map. + Deduplicated, first-seen order, so the derived rewrite map is deterministic. """ out: list[str] = [d.alias for d in enriched.dimensions] out.extend(td.alias for td in enriched.time_dimensions) diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index 2e32d9d6..6e3e205a 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -1388,15 +1388,8 @@ async def _run_and_build( ) raise timing.record("execute", _t) - # Dialect-driven read-side decode: the base hook reverses DEV-1756 - # identifier-length fitting, and BigQuery/T-SQL additionally reverse - # their alias mangling, so response keys always match SLayer's - # universal dotted shape whatever the backend did to them. - # - # The UNFILTERED alias set is passed, not ``expected_columns``: a row - # can legitimately carry a hidden ORDER-BY hoist, and the map has to be - # able to decode it. Recomputed from the pure fitting rather than - # threaded through generation. + # Reverse dialect alias fitting/mangling so response keys are canonical. + # Pass the unfiltered alias set: a row may carry a hidden ORDER-BY hoist. rows = get_dialect(prepared.dialect).decode_result_keys( rows, aliases=all_projection_aliases(prepared.enriched), ) @@ -2053,12 +2046,8 @@ async def get_column_types( logger.warning("get_column_types probe failed for model '%s'", model_name) return {} - # DEV-1756: ``sql`` was generated with length-fitted aliases, so the - # probe's metadata keys are the EMITTED names while ``em.alias`` below - # is canonical. Decode first or an over-limit measure silently drops - # out of the type map. Same hook and alias set ``_run_and_build`` uses. - # Unconditional — the hook returns its input untouched when nothing was - # shortened, so an empty ``raw_types`` needs no guard. + # The probe's metadata keys are the emitted (fitted) names; decode them + # to canonical or an over-limit measure drops out of the type map. raw_types = get_dialect(dialect).decode_result_keys( [raw_types], aliases=all_projection_aliases(enriched), )[0] @@ -3231,38 +3220,19 @@ async def _query_as_model( # NOSONAR S3776 — variable-precedence + enrich + S # from the alias by stripping the source model prefix and replacing # dots with underscores. def _alias_to_short(alias: str) -> str: - """Convert result alias to a flat column name for the virtual model. + """Flatten a result alias to the virtual model's column name, then fit it. - The query result is a self-contained table without the joins the - source model may have had, so dot syntax (join paths) is not - applicable. We use ``__`` to preserve the path information: - - 'orders.customers.regions.name' → 'customers__regions__name' - 'orders.count' → 'count' - - DEV-1756: the flattened name is then fitted to the dialect's - identifier budget. One more join hop than the example above crosses - Postgres' 63 bytes, and these names are emitted as output-column - aliases AND reused as the virtual model's ``Column.name``, so an - over-limit pair would collapse into one column. + The result is a self-contained table without joins, so join-path + dots become ``__``: 'orders.customers.regions.name' → 'customers__regions__name'. """ - # Strip source model prefix stripped = alias.split(".", 1)[-1] if "." in alias else alias - # Replace remaining dots with __ to encode the original join path return _fit_short(stripped.replace(".", "__")) def _fit_short(name: str) -> str: - """DEV-1756: bound a virtual-model short to the dialect's budget. - - Every short reaches the SQL as an output-column alias AND becomes a - ``Column.name`` downstream, so they all need this — not just the - ``_alias_to_short`` flattened ones. A measure/transform/expression - ``name``, or a user-declared cross-model rename, is supplied by the - caller and can be arbitrarily long; two such names sharing a - 63-byte prefix are silently truncated onto one column by Postgres, - and the ``short_owner`` check below cannot see it because they - differ as Python strings. Identity under the limit, so ordinary - names are untouched. + """Fit a virtual-model short to the dialect's budget; identity under the limit. + + Every short is both an output alias and a downstream ``Column.name``, + and caller-supplied names can be arbitrarily long. """ return fit_identifier( name, limit=get_dialect(dialect).max_identifier_bytes, @@ -3315,74 +3285,29 @@ def _fit_short(name: str) -> str: # leak into the virtual model's column set. Only user-declared # renames qualify for the bare-name short. if cm.user_declared and cm.name and "." not in cm.name: - # DEV-1756: still length-fitted. A rename the backend would - # truncate is not the name the user gets either way; fitting at - # least makes it deterministic and collision-checked. - short = _fit_short(cm.name) + short = _fit_short(cm.name) # still fitted: deterministic + collision-checked else: short = _alias_to_short(cm.alias) column_map.append((cm.alias, short, DataType.DOUBLE, cm.label, None, cm.format)) - # Wrap inner SQL: SELECT AS , ... FROM (inner) AS _inner - # DEV-1571 Bug 3 follow-up: identifier quoting must match the - # dialect ``inner_sql`` was generated for. On MySQL the inner CTEs - # use backticks; on T-SQL the inner CTEs use brackets AND have - # their dotted aliases mangled. Hardcoded ANSI double quotes - # would either fail to parse (MySQL) or reference an alias the - # mangled inner subquery doesn't expose (T-SQL). - # DEV-1686: the inner ``alias`` is always dialect-quoted. The ``short`` - # output alias instead has to be spelled EXACTLY as the downstream - # reference to it will be, because the two must resolve to the same - # column. So rather than approximate that policy, run the short through - # the same two mechanisms the reference side uses: - # - # * ``_maybe_quote_ident`` — DEV-1645's rule, quote iff the name - # contains an uppercase letter. This is the DEV-1756 fix: emitted - # bare, a mixed-case short was case-folded by Postgres while the - # outer stage referenced it quoted, making any query-backed model - # with a mixed-case join path or column unqueryable - # (``UndefinedColumnError``, seen on a live server). - # * ``Identifier.sql(dialect=...)`` — sqlglot adds quotes for words - # reserved IN THE TARGET DIALECT (``install_reserved_keywords`` - # unions SLayer's set into each generator's native one) and for - # names that are not safe bare identifiers. - # - # Both axes must be dialect-driven, not hardcoded. Checking only - # ``SLAYER_RESERVED_KEYWORDS`` misses MySQL-native words like ``index`` - # / ``int`` / ``rows``, which the reference side quotes and a bare - # ``AS index`` makes a syntax error; and a name like ``1abc`` or - # ``foo bar`` (``Column.name`` only forbids ``.`` and ``:``) needs - # quoting on shape alone. - # - # Quoting UNCONDITIONALLY is wrong in the other direction: it defines a - # case-sensitive ``"status"`` while the bare reference still resolves as - # ``STATUS`` on upper-folding backends (Snowflake, Oracle). + # Wrap inner SQL: SELECT AS , ... FROM (inner) AS _inner. + # The ``short`` output alias must be spelled EXACTLY as the downstream + # reference will be, so both resolve to the same column. Run it through the + # two mechanisms the reference side uses: ``_maybe_quote_ident`` (DEV-1645: + # quote iff mixed-case) and ``Identifier.sql`` (dialect-reserved words / + # unsafe shapes). Both dialect-driven — unconditional quoting breaks bare + # refs on upper-folding backends; a fixed keyword set misses native words + # like MySQL's ``index``. def _short_sql(short: str) -> str: ident = exp.Identifier(this=short, quoted=False) SQLGenerator._maybe_quote_ident(ident) return ident.sql(dialect=dialect) - # DEV-1756: the shorts share one output-column namespace, and each also - # becomes a ``Column.name`` on the virtual model. Validate the whole - # allocation keyed by the OWNING inner alias, which catches three - # things at once: two shorts that fit to the same string, two that - # differ only by case, and — the case ``_alias_to_short`` cannot - # prevent — two distinct aliases landing on the identical short because - # a user-declared cross-model ``name`` bypassed the flattening above - # and happened to match another column's canonical flat. - # - # Keyed on the CASEFOLDED short, which is deliberately CONSERVATIVE - # rather than exact. Since ``_short_sql`` quotes mixed-case shorts, - # ``Foo`` (emitted ``"Foo"``, case-preserved) and ``foo`` (emitted bare, - # folded) do in fact resolve to different columns on every dialect, so - # casefolding rejects a pair that would work. The exact rule is - # dialect-dependent — a quoted ``"EMAIL"`` collides with a bare - # ``email`` on upper-folding backends (Snowflake, Oracle) but not on - # lower-folding ones — and modelling that needs a fold-direction per - # dialect, which SLayer does not carry. Erring toward a loud error on a - # rare case-differing pair is the right side to be wrong on here: the - # alternative normalisation admits a silent duplicate output name on - # exactly the backends this machinery exists to protect. + # Validate the whole short allocation: two shorts colliding on one output + # name silently drop a column. Keyed CASEFOLDED — conservative, since a + # mixed-case short is quoted (case-preserved) and would not really collide + # with its folded twin, but SLayer carries no per-dialect fold direction, so + # a loud error beats a silent duplicate on the backends this protects. short_owner: dict[str, str] = {} for alias, short, _, _, _, _ in column_map: prior_alias = short_owner.setdefault(short.casefold(), alias) @@ -3399,14 +3324,9 @@ def _short_sql(short: str) -> str: for alias, short, _, _, _, _ in column_map ] wrapped_sql = f"SELECT {', '.join(rename_parts)} FROM ({inner_sql}) AS _inner" - # DEV-1571 Bug 2: apply the dialect's emitted-SQL rewrite (e.g. - # T-SQL bracket-mangling) so the rename clause's inner-alias - # references match what the inner subquery actually projects. - # DEV-1756: same alias set the inner SQL was generated with, so the - # wrapper's references land on the same fitted names. Safe to run over - # the combined string: ``generate()`` already replaced every canonical - # token inside ``inner_sql``, so this pass can only reach the wrapper's - # own references. + # Apply the dialect's emitted-SQL rewrite (T-SQL bracket-mangling, DEV-1756 + # length-fitting) so the wrapper's inner-alias references match the fitted + # names the inner subquery projects. Same alias set the inner SQL used. wrapped_sql = get_dialect(dialect).rewrite_emitted_sql( sql=wrapped_sql, aliases=all_projection_aliases(enriched), ) @@ -3436,8 +3356,8 @@ def _short_sql(short: str) -> str: agg_shorts.add(_alias_to_short(cm.alias)) for m in enriched.measures: if m.from_cross_model_intercept: - # DEV-1756: same fitting the ``column_map`` entry got, or the - # breadcrumb would name a column the virtual model never has. + # Same fitting the column_map entry got, else the breadcrumb + # names a column the virtual model never has. agg_shorts.add(_fit_short(m.name)) # DEV-1449: record the lineage breadcrumb so outer-stage dotted-ref diff --git a/slayer/sql/dialects/_identifier_fit.py b/slayer/sql/dialects/_identifier_fit.py index d1727bff..af8d0dd9 100644 --- a/slayer/sql/dialects/_identifier_fit.py +++ b/slayer/sql/dialects/_identifier_fit.py @@ -1,33 +1,14 @@ """DEV-1756: shared identifier-length fitting + the write-side substitution. -Backends cap identifier length, and Postgres — the tightest of the Tier-1 set -at 63 bytes — **silently truncates** past it (a NOTICE, never an error). SLayer's -universal alias convention ``..`` crosses that on -a 3-hop join, so two sibling aliases collapse onto one effective output name and -the query either fails with ``AmbiguousColumnError`` or, worse, quietly returns -a column under a name nobody looks up. - -Two primitives live here: - -:func:`fit_identifier` - Shortens an over-limit identifier to ``__``. It is a PURE - function of ``name`` — the digest covers the *full original*, not the - truncated head — which is what lets the read side rebuild the - emitted->canonical map by simply re-running it, with no map threaded through - generation. Identity when the name already fits, so the overwhelmingly - common case emits byte-identical SQL. - -:func:`substitute_quoted` - Rewrites quoted identifier tokens in emitted SQL. Driven by an exact - canonical->emitted map rather than a length regex, so — unlike a regex over - arbitrary quoted spans — the only text it can reach inside a string literal - is the exact quoted spelling of one of *this query's own* over-limit - aliases. See its docstring for the residual case. - -Sibling of :mod:`slayer.sql.dialects._alias_mangle` (the BigQuery/T-SQL dotted -alias codec) and composes with it: those dialects size against ``encode_alias`` -via the ``expand`` hook, because their mangling *lengthens* the identifier after -fitting. +Postgres SILENTLY truncates identifiers past 63 bytes (a NOTICE, never an +error), so SLayer's ``..`` aliases can collapse two +siblings onto one output name on a deep join. + +:func:`fit_identifier` shortens an over-limit name to ``__``. +It is a PURE function of ``name`` (the digest covers the full original), so the +read side rebuilds the emitted->canonical map by re-running it — no map threaded +through generation. :func:`substitute_quoted` applies that map to emitted SQL. +BigQuery/T-SQL size the budget against their post-mangle form via ``expand``. """ from __future__ import annotations @@ -36,34 +17,19 @@ from collections.abc import Callable, Mapping -#: Hex characters of digest carried in the marker. 32 bits is ample given that -#: every namespace validates its allocation and raises on collision — width only -#: affects how often that (astronomically rare) error could fire. -HASH_LEN = 8 - -#: Below this there is no room for both the marker and any readable context. -#: Every dialect SLayer supports is far above it; the guard exists so a -#: mis-configured limit fails loudly rather than emitting a useless name. -MIN_LIMIT = 16 - -#: ``_`` + digest + ``_`` -_MARKER_LEN = HASH_LEN + 2 - -#: Trimmed from the inner edges of head/tail so the marker never abuts a path -#: separator (``foo._a1b2c3d4_.bar``). -_TRIM = "._" - -#: Two-phase substitution sentinel. NUL bytes cannot appear in SQL SLayer -#: generates, so a sentinel can never be confused with real content. +HASH_LEN = 8 # digest hex chars; collisions are caught per-namespace, not by width +MIN_LIMIT = 16 # floor so a mis-configured limit fails loudly, not silently +_MARKER_LEN = HASH_LEN + 2 # ``_`` + digest + ``_`` +_TRIM = "._" # trimmed off head/tail so the marker never abuts a separator +#: NUL bytes never appear in generated SQL, so this sentinel can't clash. _SENTINEL = "\x00\x01{}\x01\x00" def _digest(name: str) -> str: - """Stable digest of the FULL original name. + """Stable digest of the full original name. - ``sha256`` rather than the builtin ``hash`` because the read side - recomputes this in a different process, where ``hash`` would be salted by - ``PYTHONHASHSEED``. Patched by tests to force collisions. + ``sha256``, not the builtin ``hash`` (salted by ``PYTHONHASHSEED`` and the + read side recomputes in another process). Patched by tests to force collisions. """ return hashlib.sha256(name.encode("utf-8")).hexdigest()[:HASH_LEN] @@ -90,23 +56,13 @@ def fit_identifier( ) -> str: """Shorten ``name`` to at most ``limit`` **bytes** as ``__``. - Returns ``name`` unchanged when ``limit`` is ``None`` (unbounded dialect) or - the name already fits — shortening only kicks in when it must, so SQL stays - readable everywhere else and existing emission tests see no churn. + Identity when ``limit`` is ``None`` or the name already fits, so common-case + SQL is byte-identical. Both ends are kept: head names the root model, tail + the column, and the repro's colliding aliases differ only in their tail. - Both ends are preserved because they carry the information: the head names - the root model, the tail names the actual column. In the reported repro the - two colliding aliases differ *only* in their final segment, so a head-only - truncation would render them indistinguishable in ``dry_run`` output. - - ``expand`` sizes the budget against a post-fit transform. BigQuery and T-SQL - mangle ``.`` to ``___`` *after* this runs, adding 2 bytes per dot; passing - ``encode_alias`` makes the loop shrink until the mangled form fits. The - returned value is NOT expanded — the dialect's own pass does that. - - Injective in practice, and any residual collision is caught by the caller's - per-namespace allocation check, so the scheme is collision-*detected* rather - than collision-free. + ``expand`` sizes the budget against a post-fit transform (BigQuery/T-SQL + ``encode_alias`` adds 2 bytes per dot); the return value itself is NOT + expanded. Residual collisions are caught by the caller's allocation check. """ grow = expand or (lambda s: s) if limit is None or len(grow(name).encode("utf-8")) <= limit: @@ -117,10 +73,8 @@ def fit_identifier( f"leave room for the {_MARKER_LEN}-byte hash marker plus context; got {limit}" ) marker = f"_{_digest(name)}_" - # Shrink the budget until the (possibly expanded) candidate fits. The final - # iteration leaves head and tail empty, yielding the bare ``__``, - # which is <= MIN_LIMIT bytes and starts with an underscore — legal even - # unquoted, where a bare hex digest could have started with a digit. + # Shrink the budget until the (expanded) candidate fits. The last iteration + # leaves head/tail empty -> bare ``__``, legal even unquoted. for budget in range(limit, _MARKER_LEN - 1, -1): avail = budget - _MARKER_LEN tail_n = avail // 2 @@ -142,26 +96,13 @@ def substitute_quoted( *, quote: Callable[[str], str], ) -> str: - """Replace each quoted ``canonical`` identifier token with its ``emitted`` - form, everywhere it appears. - - Two-phase (canonical -> sentinel -> emitted) so no substitution can be - re-read by a later one. With today's allocation the key set (over-limit) and - the value set (within-limit) are provably disjoint, so a single sequential - pass would also be correct; the two-phase form keeps that from becoming a - silent trap if the allocation ever changes. - - Only *quoted* occurrences move. A bare occurrence of the same text is a - different identifier — a table alias, say — and is left alone, which is what - keeps the deferred join-path-alias surface (DEV-1743) out of scope here. - - This is a string pass, not an AST pass, so it is not literal-aware. Being - keyed to an exact alias set rather than a length regex bounds the exposure - to one contrived case: a string literal containing the exact dialect-quoted - spelling of an over-limit alias *of the same query* (``note = '"Root.a.b. - <62 more bytes>"'``) would have its contents rewritten. A regex over quoted - spans — which the BigQuery/T-SQL dot-manglers already run over this same - SQL — has strictly wider exposure, so this pass does not add a risk class. + """Replace each quoted ``canonical`` identifier token with its ``emitted`` form. + + Two-phase (canonical -> sentinel -> emitted) so one substitution can't be + re-read by a later one. Only QUOTED occurrences move; a bare occurrence is a + different identifier (e.g. a table alias) and is left alone. String pass, not + literal-aware, but keyed to an exact alias set — no wider exposure than the + dot-mangling regexes already run over this SQL. """ if not mapping: return sql diff --git a/slayer/sql/dialects/_tier2.py b/slayer/sql/dialects/_tier2.py index 8605ff26..a90b5c54 100644 --- a/slayer/sql/dialects/_tier2.py +++ b/slayer/sql/dialects/_tier2.py @@ -34,7 +34,6 @@ class RedshiftDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = False - # DEV-1756: 127-byte identifier limit. max_identifier_bytes: int | None = 127 def build_approx_count_distinct( @@ -54,8 +53,7 @@ class TrinoDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: Trino imposes no practical identifier-length limit. - max_identifier_bytes: int | None = None + max_identifier_bytes: int | None = None # unbounded def build_approx_count_distinct( self, @@ -75,8 +73,7 @@ class PrestoDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: Presto/Athena impose no practical identifier-length limit. - max_identifier_bytes: int | None = None + max_identifier_bytes: int | None = None # unbounded def build_approx_count_distinct( self, @@ -95,8 +92,7 @@ class DatabricksDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: Databricks imposes no practical identifier-length limit. - max_identifier_bytes: int | None = None + max_identifier_bytes: int | None = None # unbounded def build_approx_count_distinct( self, @@ -115,8 +111,7 @@ class SparkDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: Spark imposes no practical identifier-length limit. - max_identifier_bytes: int | None = None + max_identifier_bytes: int | None = None # unbounded def build_approx_count_distinct( self, @@ -137,8 +132,7 @@ class OracleDialect(SqlDialect): # the canonical 2-arg LOG(base, x) form. log10_native: bool = False log2_native: bool = False - # DEV-1756: 128 bytes on 12.2+; pre-12.2 was 30 and is not modelled. - max_identifier_bytes: int | None = 128 + max_identifier_bytes: int | None = 128 # 12.2+; pre-12.2 (30) not modelled def build_approx_count_distinct( self, diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index ca29fe02..20a42915 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -186,14 +186,9 @@ class SqlDialect(BaseModel): log10_native: bool = True log2_native: bool = True - # DEV-1756: conservative universal identifier budget, in BYTES. This is - # deliberately NOT an exact model of each backend's per-identifier-class - # rules — one number that is never too generous. Bytes are conservative for - # backends that count characters (MySQL, SQL Server). ``None`` means - # effectively unbounded, in which case every fitting hook is a no-op. - # The base default is the tightest Tier-1 value (Postgres' NAMEDATALEN-1), - # so a dialect added later without setting it over-shortens rather than - # silently truncating. + # Conservative universal identifier budget in BYTES; ``None`` = unbounded + # (fitting hooks become no-ops). Default is the tightest Tier-1 value + # (Postgres), so a new dialect over-shortens rather than silently truncating. max_identifier_bytes: int | None = 63 @property @@ -504,43 +499,34 @@ def emit_outer_wrap( out += "\n" + offset_arg.sql(dialect=self.sqlglot_name, pretty=True) return out - # ------------------------------------------------------------------ - # DEV-1756: identifier-length fitting - # - # Postgres truncates over-limit identifiers SILENTLY, so two long sibling - # aliases collapse onto one output name. Aliases stay canonical everywhere - # inside SLayer; they are fitted only on emission and restored on the - # result keys, so consumers never observe the dialect dependence. - # ------------------------------------------------------------------ + # DEV-1756 identifier-length fitting. Aliases stay canonical inside SLayer, + # fitted only on emission and restored on the result keys. def quote_identifier(self, name: str) -> str: """``name`` wrapped in this dialect's identifier quotes.""" return exp.Identifier(this=name, quoted=True).sql(dialect=self.sqlglot_name) def fit_alias(self, name: str) -> str: - """LENGTH-ONLY fitting. Identity when ``name`` already fits. + """Length-only fitting; identity when ``name`` already fits. - This — not ``emit_alias`` — drives the write pass, which is why an - under-limit alias produces byte-identical SQL on every dialect, - including the ones that separately mangle dots. + Drives the write pass (not ``emit_alias``), so an under-limit alias + produces byte-identical SQL even on dialects that mangle dots. """ return fit_identifier(name=name, limit=self.max_identifier_bytes) def emit_alias(self, alias: str) -> str: - """The FINAL identifier a canonical alias reaches the SQL as. + """The final identifier a canonical alias reaches the SQL as. - Equals ``fit_alias`` here; ``BigqueryDialect`` / ``TsqlDialect`` - compose their dot-mangling on top. Used to build the read-side map, so - it must match the emitted token exactly. + Equals ``fit_alias`` here; BigQuery/T-SQL compose dot-mangling on top. + Used to build the read-side map, so must match the emitted token exactly. """ return self.fit_alias(alias) def alias_rewrite_map(self, aliases: Sequence[str]) -> dict[str, str]: """``{canonical: fitted}`` for the write pass, only where they differ. - The collision check covers EVERY alias including the identities: an - already-short alias whose spelling equals another's fitted form is just - as much a duplicate output name, and no hash width can prevent it. + The collision check covers every alias including identities: a short + alias equal to another's fitted form is just as much a duplicate. """ if self.max_identifier_bytes is None: return {} @@ -562,9 +548,8 @@ def alias_rewrite_map(self, aliases: Sequence[str]) -> dict[str, str]: return {k: v for k, v in allocation.items() if k != v} def decode_alias_map(self, aliases: Sequence[str]) -> dict[str, str]: - """``{emitted: canonical}`` — the read-side inverse, rebuilt by simply - re-running the (pure) fitting rather than threading a map through - generation.""" + """``{emitted: canonical}`` — read-side inverse, rebuilt by re-running + the pure fitting rather than threading a map through generation.""" out: dict[str, str] = {} for alias in aliases: emitted = self.emit_alias(alias) @@ -579,14 +564,12 @@ def _rekey_row( *, fallback: Callable[[str], str] | None = None, ) -> dict[str, Any]: - """Apply ``mapping`` to one row's keys, refusing to let two keys - collapse onto one (which would silently drop a column's values). - - ``fallback`` decodes keys absent from ``mapping`` — BigQuery and T-SQL - pass their ``___`` -> ``.`` bijection. It must be applied HERE rather - than in a separate dict comprehension upstream: pre-decoding into a - dict would let two keys collapse before this check ever ran, which is - precisely the silent-column-loss this class exists to prevent. + """Apply ``mapping`` to a row's keys, erroring if two keys collapse onto + one (silent column loss). + + ``fallback`` decodes keys absent from ``mapping`` (BigQuery/T-SQL + ``___`` -> ``.``). Applied here, not upstream, so the collapse check + sees every key. """ out: dict[str, Any] = {} for key, value in row.items(): @@ -608,28 +591,14 @@ def _rekey_row( def rewrite_emitted_sql( self, sql: str, *, aliases: Sequence[str] = (), ) -> str: - """Post-pass string-level rewrite of the final generator output. - - Symmetric companion to ``rewrite_parsed_ast`` (the input-side hook): - write-side, applied at the end of ``SQLGenerator.generate()`` AFTER - ``_apply_outer_projection_trim``. - - Base impl performs the DEV-1756 length pass: each canonical alias in - ``aliases`` whose fitted form differs has its dialect-quoted token - replaced, everywhere it occurs (inner ``AS``, outer wrap projection, - ORDER BY, CTE column references). Driven by the query's own alias set - rather than a length regex, which bounds what it can touch inside a - string literal — see :func:`substitute_quoted`. - - ``aliases`` defaults to empty, which makes this a no-op — every caller - that does not supply an alias set keeps today's behaviour exactly. - - Contract: preserve query semantics. Suitable for alias renames, - identifier mangling/escape, dialect-quoting fixes. Do NOT change query - shape — use the typed ``build_*`` methods on this class for that. - - Overrides today: ``BigqueryDialect`` / ``TsqlDialect`` compose their - dotted-alias mangling AFTER this length pass. + """Post-pass string rewrite of the final SQL; write-side companion to + ``rewrite_parsed_ast``, applied at the end of ``generate()``. + + Base impl fits over-limit aliases (DEV-1756), replacing each canonical + token everywhere it occurs. Driven by the query's own alias set, not a + length regex, which bounds what it can touch (see :func:`substitute_quoted`). + Empty ``aliases`` is a no-op. Must preserve query semantics, not shape. + BigQuery/T-SQL compose dot-mangling after this pass. """ mapping = self.alias_rewrite_map(aliases) if not mapping: @@ -642,15 +611,10 @@ def decode_result_keys( *, aliases: Sequence[str] = (), ) -> list[dict[str, Any]]: - """Reverse-pass on result-row keys to undo the write-side rewrite. - - Called at the end of ``SlayerQueryEngine.execute()`` so consumers - always see SLayer's universal alias shape (``orders._count``, - ``orders.products.category``) regardless of which dialect a query ran - on — and regardless of whether its aliases had to be shortened. + """Reverse the write-side rewrite on result-row keys, so consumers always + see SLayer's canonical alias shape regardless of dialect or shortening. - Overrides today: ``BigqueryDialect`` / ``TsqlDialect`` additionally - decode the ``___`` mangling back to dots. + BigQuery/T-SQL additionally decode the ``___`` mangling back to dots. """ mapping = self.decode_alias_map(aliases) if not mapping: diff --git a/slayer/sql/dialects/bigquery.py b/slayer/sql/dialects/bigquery.py index d640125a..d641d132 100644 --- a/slayer/sql/dialects/bigquery.py +++ b/slayer/sql/dialects/bigquery.py @@ -89,8 +89,7 @@ class BigqueryDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: 300-character column-name limit. - max_identifier_bytes: int | None = 300 + max_identifier_bytes: int | None = 300 # column-name limit def build_approx_count_distinct( self, @@ -134,13 +133,8 @@ def build_date_trunc( ) def fit_alias(self, name: str) -> str: - """DEV-1756: size the length budget against the POST-mangle form. - - ``rewrite_emitted_sql`` expands every ``.`` to ``___`` after fitting, - adding 2 bytes per dot, so fitting to the raw limit would bust it on a - deep chain. The returned value is still dotted — the regex below does - the mangling. - """ + """Size the budget against the post-mangle form (``.`` -> ``___`` adds 2 + bytes per dot); return value stays dotted for the regex below.""" return fit_identifier( name=name, limit=self.max_identifier_bytes, expand=encode_alias, ) @@ -152,22 +146,12 @@ def emit_alias(self, alias: str) -> str: def rewrite_emitted_sql( self, sql: str, *, aliases: Sequence[str] = (), ) -> str: - """Replace ``.`` with ``___`` inside backtick-quoted identifiers. - - Applied as a post-pass on the BigQuery dialect's final SQL so - emitted column aliases (``SELECT ... AS \\`orders._count\\``) and - references to those aliases - (``ORDER BY \\`orders._count\\``) comply with BigQuery's column-name - grammar. - - DEV-1756: the base class's LENGTH pass runs first. An under-limit alias - is untouched by it (``fit_alias`` is the identity), so the regex below - sees exactly what it sees today and the output stays byte-identical. - An over-limit alias is rewritten to ``__`` whose - head/tail are still dotted, so the regex mangles it here — yielding - ``encode_alias(fit_alias(a))``, which is what ``emit_alias`` returns. - Because the fitted form is mangled by this same pass rather than - arriving pre-mangled, there is no double-encoding. + """Replace ``.`` with ``___`` inside backtick-quoted identifiers, so + emitted aliases and their references satisfy BigQuery's grammar. + + The base LENGTH pass runs first: it no-ops on under-limit aliases (SQL + stays byte-identical) and rewrites over-limit ones to a still-dotted + form that this regex then mangles — no double-encoding. """ sql = super().rewrite_emitted_sql(sql=sql, aliases=aliases) return _DOTTED_ALIAS_RE.sub( @@ -180,16 +164,11 @@ def decode_result_keys( *, aliases: Sequence[str] = (), ) -> list[dict[str, Any]]: - """Reverse the BigQuery alias mangling on result-row keys so - consumers see SLayer's universal dotted alias shape regardless of - whether the query ran against BigQuery or another dialect. - - DEV-1756: keys produced by a length-fitted alias are not recoverable - from the key alone, so the ``emitted -> canonical`` map is consulted - first; anything outside it falls back to the pure ``___`` -> ``.`` - bijection, preserving today's behaviour for short aliases. Both steps - happen inside ``_rekey_row`` in ONE pass — pre-decoding into a dict - first would let two keys collapse before the duplicate check ran. + """Reverse the BigQuery alias mangling on result-row keys so consumers + see SLayer's universal dotted shape whatever dialect ran the query. + + Fitted keys aren't recoverable alone, so the ``emitted -> canonical`` + map is consulted first, falling back to the ``___`` -> ``.`` bijection. """ mapping = self.decode_alias_map(aliases) return [ diff --git a/slayer/sql/dialects/clickhouse.py b/slayer/sql/dialects/clickhouse.py index 7eab146c..4ae4cf0a 100644 --- a/slayer/sql/dialects/clickhouse.py +++ b/slayer/sql/dialects/clickhouse.py @@ -21,8 +21,7 @@ class ClickhouseDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: ClickHouse imposes no practical identifier-length limit. - max_identifier_bytes: int | None = None + max_identifier_bytes: int | None = None # unbounded def build_median( self, diff --git a/slayer/sql/dialects/duckdb.py b/slayer/sql/dialects/duckdb.py index 8b4eba52..576690fc 100644 --- a/slayer/sql/dialects/duckdb.py +++ b/slayer/sql/dialects/duckdb.py @@ -21,8 +21,7 @@ class DuckdbDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: DuckDB accepts long identifiers; 256 is a safe documented ceiling. - max_identifier_bytes: int | None = 256 + max_identifier_bytes: int | None = 256 # safe documented ceiling def build_approx_count_distinct( self, diff --git a/slayer/sql/dialects/mysql.py b/slayer/sql/dialects/mysql.py index c861d433..04dec814 100644 --- a/slayer/sql/dialects/mysql.py +++ b/slayer/sql/dialects/mysql.py @@ -25,7 +25,7 @@ class MysqlDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: 64-char identifier limit. MySQL allows 256 for column ALIASES specifically, but one conservative number keeps the rule simple, and MySQL errors rather than truncating. + # Conservative: MySQL allows 256 for column aliases but errors (not truncates). max_identifier_bytes: int | None = 64 def build_median( diff --git a/slayer/sql/dialects/postgres.py b/slayer/sql/dialects/postgres.py index 4868d7e8..3562df9e 100644 --- a/slayer/sql/dialects/postgres.py +++ b/slayer/sql/dialects/postgres.py @@ -45,7 +45,7 @@ class PostgresDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: NAMEDATALEN 64 -> 63 usable bytes; over-length names are SILENTLY truncated. + # NAMEDATALEN: 63 usable bytes; over-length names are SILENTLY truncated. max_identifier_bytes: int | None = 63 def rewrite_target_ast(self, tree: exp.Expression) -> exp.Expression: diff --git a/slayer/sql/dialects/snowflake.py b/slayer/sql/dialects/snowflake.py index fbe89f8b..08590557 100644 --- a/slayer/sql/dialects/snowflake.py +++ b/slayer/sql/dialects/snowflake.py @@ -193,7 +193,6 @@ class so ``engine_factory`` / ``client`` stay dialect-agnostic. log10_native: bool = True # No native LOG2 — falls through to canonical ``LOG(2, x)`` form. log2_native: bool = False - # DEV-1756: 255-character identifier limit. max_identifier_bytes: int | None = 255 def build_approx_count_distinct( diff --git a/slayer/sql/dialects/sqlite.py b/slayer/sql/dialects/sqlite.py index 9b66059c..4569279a 100644 --- a/slayer/sql/dialects/sqlite.py +++ b/slayer/sql/dialects/sqlite.py @@ -406,8 +406,7 @@ class SqliteDialect(SqlDialect): explain_postfix: str = "" log10_native: bool = True log2_native: bool = True - # DEV-1756: SQLite imposes no identifier-length limit. - max_identifier_bytes: int | None = None + max_identifier_bytes: int | None = None # unbounded def build_date_trunc( self, diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index 29d15b85..a435bed7 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -71,8 +71,7 @@ class TsqlDialect(SqlDialect): explain_postfix: str = "; SET SHOWPLAN_ALL OFF" log10_native: bool = True log2_native: bool = False - # DEV-1756: sysname is nvarchar(128), i.e. 128 characters. - max_identifier_bytes: int | None = 128 + max_identifier_bytes: int | None = 128 # sysname is nvarchar(128) def build_approx_count_distinct( self, @@ -338,12 +337,8 @@ def emit_outer_wrap( # ------------------------------------------------------------------ def fit_alias(self, name: str) -> str: - """DEV-1756: size the length budget against the POST-mangle form. - - ``rewrite_emitted_sql`` expands every ``.`` to ``___`` after fitting, - adding 2 bytes per dot, so fitting to the raw 128-byte limit would bust - it on a deep chain. The returned value is still dotted. - """ + """Size the budget against the post-mangle form (``.`` -> ``___`` adds 2 + bytes per dot); return value stays dotted for the regex below.""" return fit_identifier( name=name, limit=self.max_identifier_bytes, expand=encode_alias, ) @@ -357,24 +352,12 @@ def rewrite_emitted_sql( ) -> str: """Replace ``.`` with ``___`` inside bracket-quoted identifiers. - T-SQL's ``ORDER BY`` resolver does not treat ``[a.b]`` as a - SELECT alias — it tries to resolve it as a column-name lookup - against the FROM scope and fails with ``Invalid column name``. - Mangling on emit gives the parser a single dotless identifier - and the alias resolves cleanly. ``decode_result_keys`` reverses - the mangling on result rows so consumers see SLayer's universal - dotted alias shape. - - Uses the same bijection as ``BigqueryDialect`` (shared encode in - ``slayer.sql.dialects._alias_mangle``); only the regex anchor - differs. - - DEV-1756: the base class's LENGTH pass runs first. An under-limit alias - is untouched by it (``fit_alias`` is the identity), so the regex below - sees exactly what it sees today and the output stays byte-identical. - An over-limit alias arrives as ``__`` with head/tail - still dotted, so this pass mangles it — yielding what ``emit_alias`` - returns, with no double-encoding. + T-SQL's ``ORDER BY`` resolver treats ``[a.b]`` as a column lookup, not a + SELECT alias, and fails; a dotless identifier resolves cleanly. Same + bijection as ``BigqueryDialect``, only the regex anchor differs. + + The base LENGTH pass runs first: no-op on under-limit aliases, and + over-limit ones arrive still-dotted for this pass — no double-encoding. """ sql = super().rewrite_emitted_sql(sql=sql, aliases=aliases) return _TSQL_DOTTED_ALIAS_RE.sub( @@ -387,16 +370,11 @@ def decode_result_keys( *, aliases: Sequence[str] = (), ) -> list[dict[str, Any]]: - """Reverse the T-SQL alias mangling on result-row keys so - consumers see SLayer's universal dotted alias shape regardless - of whether the query ran against T-SQL or another dialect. - - DEV-1756: keys produced by a length-fitted alias are not recoverable - from the key alone, so the ``emitted -> canonical`` map is consulted - first; anything outside it falls back to the pure ``___`` -> ``.`` - bijection, preserving today's behaviour for short aliases. Both steps - happen inside ``_rekey_row`` in ONE pass — pre-decoding into a dict - first would let two keys collapse before the duplicate check ran. + """Reverse the T-SQL alias mangling on result-row keys so consumers see + SLayer's universal dotted shape whatever dialect ran the query. + + Fitted keys aren't recoverable alone, so the ``emitted -> canonical`` + map is consulted first, falling back to the ``___`` -> ``.`` bijection. """ mapping = self.decode_alias_map(aliases) return [ diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index fc7f5a67..8489e099 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -268,19 +268,12 @@ def _parse_window_duration(value: str) -> list[tuple[int, str]]: def _cte_name_from_alias(prefix: str, alias: str, *, limit: int | None = None) -> str: """Build a unique CTE name from a measure alias. - Dots are replaced with ``__`` (double underscore) to avoid collision - with aliases that already contain underscores. E.g.: - - ``orders.revenue_sum`` -> ``_fm_orders__revenue_sum`` - - ``orders_v2.revenue_sum`` -> ``_fm_orders_v2__revenue_sum`` - - DEV-1756: ``limit`` is the dialect's ``max_identifier_bytes``. The whole - result is fitted, PREFIX INCLUDED — a ``cp_value_12_`` prefix eats 12 of - Postgres' 63 bytes before the alias is even considered. CTE names are - emitted unquoted, so an over-limit definition and its reference would be - truncated to the same thing by the server today (harmless) or to a - collision with a sibling CTE (not harmless). Stays a pure function of - ``(prefix, alias, limit)``, so every call site derives the same name and - definition and reference cannot drift. + Dots become ``__`` to avoid colliding with aliases that already contain + underscores: ``orders.revenue_sum`` -> ``_fm_orders__revenue_sum``. + + The whole result is fitted to ``limit`` (the dialect's max), prefix + included. Pure function of ``(prefix, alias, limit)``, so a CTE's definition + and every reference derive the same name. """ sanitized = alias.replace(".", "__") sanitized = re.sub(r"[^a-zA-Z0-9_]", "_", sanitized) @@ -418,25 +411,16 @@ def __init__(self, dialect: str | SqlDialect = "postgres"): self._dialect: SqlDialect = dialect else: self._dialect = get_dialect(dialect) - # DEV-1756: CTE-name allocation for the statement being generated, - # ``emitted -> (prefix, alias)``. Reset per ``generate()``. + # Per-statement CTE-name allocation, ``emitted -> (prefix, alias)``. self._cte_names: dict[str, tuple[str, str]] = {} def _cte_name(self, prefix: str, alias: str) -> str: """Allocate a length-fitted CTE name, refusing a collision. - DEV-1756: CTE names live in one namespace per statement and are emitted - UNQUOTED, so two that fit to the same string would produce a query that - silently references the wrong CTE. Repeat calls with the same - ``(prefix, alias)`` are the same name, not a collision — several code - paths re-derive a CTE's name to reference it. - - Keyed CASEFOLDED, because unquoted is exactly the case that the server - folds: ``_wm_Foo`` and ``_wm_foo`` are one identifier on Postgres (and - on Snowflake), so comparing the literal spellings would wave through a - pair that silently resolves to the same CTE. Unlike the projection - aliases, there is no quoted variant to exempt here — these are never - quoted — so casefolding is exact rather than merely conservative. + CTE names are emitted UNQUOTED, so two that fit to the same string + silently reference the wrong CTE. Keyed CASEFOLDED, since the server + folds unquoted names. Repeat calls with the same ``(prefix, alias)`` + return the same name, not a collision. """ name = _cte_name_from_alias( prefix=prefix, alias=alias, limit=self._dialect.max_identifier_bytes, @@ -643,8 +627,7 @@ def generate( raise ValueError( f"render_mode must be 'outer' or 'wrapped', got {render_mode!r}" ) - # DEV-1756: CTE names are allocated per statement. - self._cte_names = {} + self._cte_names = {} # reset per-statement CTE-name allocation has_isolated = any(_has_cross_model_filter(m) for m in enriched.measures) has_windowed = any(_is_windowed_measure(m) for m in enriched.measures) has_cross_model = bool(enriched.cross_model_measures) @@ -675,16 +658,9 @@ def generate( if render_mode == "outer": sql = self._apply_outer_projection_trim(sql=sql, enriched=enriched) - # Dialect-driven post-pass: the base class fits over-limit projection - # aliases to the dialect's identifier budget (DEV-1756); BigQuery and - # T-SQL additionally mangle dotted aliases. Fires for BOTH render modes - # — inner CTE column names are subject to the same dialect alias rules - # as the outer projection. - # - # The alias set is the UNFILTERED one: hidden ORDER-BY hoists and - # ``_inner_*`` / ``_ft*`` / ``_ts*`` entries are projected in the inner - # SELECT and truncate exactly like user-declared aliases, so a filtered - # list would leave their references pointing at an unfitted name. + # Dialect post-pass: fit over-limit aliases (DEV-1756), plus BigQuery/T-SQL + # dot-mangling. Fires for both render modes. Unfiltered alias set: hidden + # hoists are projected too and must be fitted with the same map. sql = self._dialect.rewrite_emitted_sql( sql=sql, aliases=all_projection_aliases(enriched), ) @@ -1688,10 +1664,8 @@ def _generate_with_computed(self, enriched: EnrichedQuery, for t in deferred_self_joins: src_cte = ctes[-1][0] - # DEV-1756: these are CTE names built from a USER-supplied - # transform name, so they need the same length fitting and - # collision check as every other CTE. Allocated once and reused - # for the definition and every reference below. + # Built from a user-supplied transform name; fit + collision-check + # like every CTE. Allocated once, reused for def and references. shift_name = self._cte_name(prefix="shifted_", alias=t.name) shifted_sql = self._generate_shifted_base( enriched=enriched, transform=t, diff --git a/tests/dialects/test_identifier_fit.py b/tests/dialects/test_identifier_fit.py index 90c88bc0..5785d4b0 100644 --- a/tests/dialects/test_identifier_fit.py +++ b/tests/dialects/test_identifier_fit.py @@ -1,27 +1,8 @@ """DEV-1756: the identifier-length primitive and its dialect wiring. -Postgres' NAMEDATALEN is 64, so identifiers are capped at 63 BYTES and anything -longer is SILENTLY truncated (a NOTICE, never an error). SLayer's projection -aliases (``..``) cross that on a 3-hop join, so two -sibling aliases can collapse onto one effective output name. - -``fit_identifier`` is the shared primitive that shortens an over-limit -identifier to a deterministic ``__``. It is a PURE function -of ``name`` (the digest covers the full original), which is what lets the read -side rebuild the emitted->canonical map without threading anything through -generation. - -``substitute_quoted`` is the write-side primitive: a TWO-PHASE replacement -(canonical -> sentinel -> final) so no substitution can be re-read by a later -one. Today the key set (over-limit) and the value set (within-limit) are -provably disjoint, so a naive sequential replace would also be correct — the -two-phase form is defence against that invariant being weakened later, and is -tested directly rather than through the alias API where the disjointness makes -a cascade unconstructible. - -Emission/behaviour tests for the three surfaces live in -``tests/test_dev1756_identifier_length.py``; live execution is in the Postgres -integration suite. +Postgres caps identifiers at 63 bytes and silently truncates longer ones, so sibling +aliases can collapse. ``fit_identifier`` shortens to a pure ``__``; +``substitute_quoted`` is the two-phase write side. Emission: ``test_dev1756_identifier_length.py``. """ from __future__ import annotations @@ -37,8 +18,6 @@ from slayer.core.errors import IdentifierCollisionError from slayer.sql.dialects import _ALL_DIALECTS, get_dialect from slayer.sql.dialects._alias_mangle import encode_alias - -# The feature under test. from slayer.sql.dialects._identifier_fit import ( HASH_LEN, MIN_LIMIT, @@ -47,16 +26,14 @@ ) -# The DEV-1756 repro pair: 73 and 74 bytes, sharing a 63-byte prefix. +# Repro pair: 73 and 74 bytes, sharing a 63-byte prefix. LONG_NAME = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name" LONG_EMAIL = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email" -# Two over-limit names that differ ONLY in the middle, so head and tail both -# survive fitting identically — the shape needed to force a digest collision. +# Differ only in the middle, so head and tail survive fitting identically (forces a digest collision). TWIN_A = "SandboxAlpha." * 3 + "111" + ".SandboxOmega" * 3 TWIN_B = "SandboxAlpha." * 3 + "222" + ".SandboxOmega" * 3 -# Every limit SLayer configures on a dialect, plus Postgres' binding 63. ALL_LIMITS = (63, 64, 127, 128, 255, 256, 300) _UNQUOTED_IDENT_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") @@ -72,11 +49,6 @@ def _dq(name: str) -> str: return f'"{name}"' -# --------------------------------------------------------------------------- -# fit_identifier — core contract -# --------------------------------------------------------------------------- - - class TestFitIdentifierCore: def test_repro_pair_is_actually_over_the_postgres_limit(self) -> None: """Guard the premise: without this the rest of the file is vacuous.""" @@ -85,7 +57,6 @@ def test_repro_pair_is_actually_over_the_postgres_limit(self) -> None: assert LONG_NAME.encode()[:63] == LONG_EMAIL.encode()[:63] def test_under_limit_returned_unchanged(self) -> None: - """The common path must be a true identity — no hash, no allocation.""" assert fit_identifier("orders.revenue_sum", limit=63) == "orders.revenue_sum" def test_exactly_at_limit_returned_unchanged(self) -> None: @@ -97,7 +68,6 @@ def test_one_byte_over_limit_is_shortened(self) -> None: assert fit_identifier(name, limit=63) != name def test_none_limit_is_a_no_op(self) -> None: - """Unbounded dialects (SQLite/ClickHouse/Trino/...) never shorten.""" assert fit_identifier(LONG_EMAIL, limit=None) == LONG_EMAIL @pytest.mark.parametrize("limit", ALL_LIMITS) @@ -106,9 +76,7 @@ def test_output_within_limit_bytes(self, limit: int) -> None: assert _nbytes(fit_identifier(name, limit=limit)) <= limit def test_output_is_pinned_exactly(self) -> None: - """Pin the whole result, not `f(x) == f(x)` — which proves nothing - about a pure function. Also documents the head/tail split concretely: - 27 head bytes, the 10-byte marker, 26 tail bytes.""" + """Pin the whole result: 27 head bytes, 10-byte marker, 26 tail bytes.""" expected = ( "SandboxInvoiceV2.SandboxSub" f"_{hashlib.sha256(LONG_EMAIL.encode()).hexdigest()[:HASH_LEN]}_" @@ -118,8 +86,6 @@ def test_output_is_pinned_exactly(self) -> None: assert _nbytes(expected) == 63 def test_marker_is_exactly_sha256_of_the_full_original(self) -> None: - """Pin the digest's ALGORITHM, POSITION and INPUT — not merely that - eight hex characters appear somewhere.""" got = fit_identifier(LONG_EMAIL, limit=63) match = _MARKER_RE.search(got) assert match, got @@ -127,8 +93,7 @@ def test_marker_is_exactly_sha256_of_the_full_original(self) -> None: assert match.group(1) == expected def test_digest_is_not_process_dependent(self) -> None: - """The read side recomputes the map in a DIFFERENT process, so the - digest must not depend on PYTHONHASHSEED (i.e. not builtin ``hash``).""" + """The read side recomputes in another process, so the digest must not depend on PYTHONHASHSEED.""" env = os.environ.copy() env["PYTHONHASHSEED"] = "12345" out = subprocess.run( @@ -142,15 +107,12 @@ def test_digest_is_not_process_dependent(self) -> None: assert out.stdout.strip() == fit_identifier(LONG_EMAIL, limit=63) def test_head_and_tail_both_preserved(self) -> None: - """Readability contract: the root model AND the leaf column survive, - which is what makes colliding siblings tellable apart in dry_run SQL.""" got = fit_identifier(LONG_EMAIL, limit=63) assert got.startswith("SandboxInvoiceV2") assert got.endswith("email") def test_repro_siblings_differ_outside_the_hash(self) -> None: - """The two aliases that collide on Postgres must stay distinguishable - by eye, not only by digest.""" + """Colliding aliases stay distinguishable by eye, not only by digest.""" a = fit_identifier(LONG_NAME, limit=63) b = fit_identifier(LONG_EMAIL, limit=63) assert a != b @@ -167,11 +129,8 @@ def test_shape_is_head_underscore_hash_underscore_tail(self) -> None: assert _MARKER_RE.search(got), got def test_separators_are_trimmed_next_to_the_marker(self) -> None: - """``head.rstrip("._")`` / ``tail.lstrip("._")`` — otherwise a budget - cut landing on a path separator yields ``foo._a1b2c3d4_.bar``.""" - # 'a.' repeated: every even byte offset lands on a '.', so an untrimmed - # head/tail would abut the marker with a separator. - name = "a." * 60 + """Head/tail are stripped of ``._`` so a cut on a separator won't yield ``foo._a1b2c3d4_.bar``.""" + name = "a." * 60 # every even offset lands on a '.' got = fit_identifier(name, limit=63) match = _MARKER_RE.search(got) assert match, got @@ -180,24 +139,17 @@ def test_separators_are_trimmed_next_to_the_marker(self) -> None: assert not tail.startswith((".", "_")), got def test_minimum_budget_form_is_legal(self) -> None: - """At the tightest legal budget the head/tail collapse away and the - result is the bare ``__`` marker — which must still be a legal - leading character (a bare hex digest can start with a digit).""" + """At the tightest budget the result is the bare ``__`` marker, still a legal identifier.""" got = fit_identifier(LONG_EMAIL, limit=MIN_LIMIT) assert _nbytes(got) <= MIN_LIMIT assert _UNQUOTED_IDENT_RE.match(got.replace(".", "_")), got -# --------------------------------------------------------------------------- -# fit_identifier — edge cases -# --------------------------------------------------------------------------- - - class TestFitIdentifierEdges: def test_multibyte_never_splits_a_codepoint(self) -> None: name = "é" * 60 # 120 bytes of 2-byte codepoints got = fit_identifier(name, limit=63) - got.encode("utf-8").decode("utf-8") # would raise if a codepoint split + got.encode("utf-8").decode("utf-8") # raises if a codepoint was split assert _nbytes(got) <= 63 def test_multibyte_bound_is_bytes_not_characters(self) -> None: @@ -206,16 +158,14 @@ def test_multibyte_bound_is_bytes_not_characters(self) -> None: assert _nbytes(got) <= 63 < len(name) * 2 def test_unquoted_legality_preserved_for_flat_input(self) -> None: - """Surfaces 3 (CTE names) and 4 (virtual-model shorts) are emitted - UNQUOTED, so a fitted flat name must stay a legal bare identifier.""" + """CTE names and virtual-model shorts are emitted unquoted, so a fitted flat name stays legal.""" flat = "_cm_" + "SandboxSubscription__SandboxCustomer__SandboxConsumer__" * 2 got = fit_identifier(flat, limit=63) assert _UNQUOTED_IDENT_RE.match(got), got @pytest.mark.parametrize("limit", range(MIN_LIMIT, 40)) def test_never_starts_with_a_digit(self, limit: int) -> None: - """A bare hex digest can begin with a digit, which is illegal unquoted - on several dialects.""" + """A bare hex digest can begin with a digit, illegal unquoted on several dialects.""" assert not fit_identifier(LONG_EMAIL, limit=limit)[0].isdigit() @pytest.mark.parametrize("limit", range(MIN_LIMIT, 40)) @@ -223,8 +173,6 @@ def test_tiny_budget_still_within_limit(self, limit: int) -> None: assert _nbytes(fit_identifier(LONG_EMAIL, limit=limit)) <= limit def test_head_trimmed_to_empty_still_legal(self) -> None: - """A name whose head budget lands entirely inside separators must not - yield a leading-separator-then-digit mess.""" name = "." * 40 + "abcdefghij" * 5 got = fit_identifier(name, limit=MIN_LIMIT) assert _nbytes(got) <= MIN_LIMIT @@ -239,23 +187,18 @@ def test_limit_below_minimum_raises(self) -> None: fit_identifier(LONG_EMAIL, limit=MIN_LIMIT - 1) -# --------------------------------------------------------------------------- # fit_identifier — the `expand` hook (BigQuery / T-SQL dot-mangling) -# --------------------------------------------------------------------------- class TestFitIdentifierExpand: def test_post_mangle_length_within_limit(self) -> None: - """BigQuery/T-SQL mangle `.` -> `___` AFTER fitting, adding 2 bytes per - dot. Fitting to the raw limit would then bust it, so the budget must be - computed against the expanded form.""" + """Budget is computed against the expanded form since BigQuery/T-SQL mangle ``.`` -> ``___`` after fitting.""" name = ".".join(["Sandbox" * 4] * 6) # many dots, well over 128 got = fit_identifier(name, limit=128, expand=encode_alias) assert _nbytes(encode_alias(got)) <= 128 def test_expand_is_not_applied_to_the_return_value(self) -> None: - """``fit_identifier`` only SIZES against the expansion; the dialect's - own regex performs the actual mangling.""" + """``fit_identifier`` only sizes against the expansion; the dialect does the mangling.""" name = ".".join(["Sandbox" * 4] * 6) got = fit_identifier(name, limit=128, expand=encode_alias) assert "___" not in got @@ -264,8 +207,7 @@ def test_expand_under_limit_is_identity(self) -> None: assert fit_identifier("a.b", limit=128, expand=encode_alias) == "a.b" def test_aggressively_expanding_transform_still_fits(self) -> None: - """The budget loop must keep shrinking until the EXPANDED form fits, - even when the expansion is far more aggressive than dot-mangling.""" + """The budget loop shrinks until the expanded form fits, however aggressive the expansion.""" def explode(s: str) -> str: return s + "#" * (40 * s.count(".")) @@ -274,9 +216,7 @@ def explode(s: str) -> str: assert _nbytes(explode(got)) <= 63 -# --------------------------------------------------------------------------- # substitute_quoted — the write-side primitive (two-phase) -# --------------------------------------------------------------------------- class TestSubstituteQuoted: @@ -291,9 +231,7 @@ def test_replaces_every_occurrence(self) -> None: assert '"a.b"' not in got def test_chained_mapping_does_not_cascade(self) -> None: - """THE two-phase requirement: with ``A -> B`` and ``B -> C`` in one - map, the ``A`` occurrence must land on ``B`` and STOP. A naive - sequential ``str.replace`` would carry it on to ``C``.""" + """Two-phase pass: ``A->B`` and ``B->C`` must not cascade A into C.""" sql = 'SELECT "A", "B"' got = substitute_quoted(sql, {"A": "B", "B": "C"}, quote=_dq) assert got == 'SELECT "B", "C"' @@ -305,44 +243,32 @@ def test_chained_mapping_is_order_independent(self) -> None: assert forward == reverse == 'SELECT "B", "C"' def test_swap_is_not_a_cascade(self) -> None: - """``A -> B`` and ``B -> A`` simultaneously — only a two-phase pass - gets this right.""" + """Simultaneous ``A->B`` and ``B->A`` — only a two-phase pass gets this right.""" got = substitute_quoted('SELECT "A", "B"', {"A": "B", "B": "A"}, quote=_dq) assert got == 'SELECT "B", "A"' def test_only_quoted_occurrences_are_replaced(self) -> None: - """A bare (unquoted) occurrence of the same text is a different - identifier — a table alias, say — and must be left alone.""" + """A bare occurrence of the same text (a table alias, say) is left alone.""" sql = 'SELECT tbl.col AS "tbl.col" FROM t' got = substitute_quoted(sql, {"tbl.col": "z"}, quote=_dq) assert got == 'SELECT tbl.col AS "z" FROM t' def test_does_not_reach_into_string_literals(self) -> None: - """The pass is keyed on exact quoted tokens, never on a length regex, - so a long run of text between two double quotes inside a literal is - untouched.""" + """Keyed on exact quoted tokens, not a length regex, so quoted text inside a literal is untouched.""" literal = "x" * 90 sql = f'SELECT 1 AS "a.b" WHERE note LIKE \'%"{literal}"%\'' got = substitute_quoted(sql, {"a.b": "z"}, quote=_dq) assert f'"{literal}"' in got def test_sentinel_cannot_leak_into_the_output(self) -> None: - """Whatever sentinel the two-phase pass uses must not survive, even if - the SQL happens to contain sentinel-looking text.""" + """The two-phase sentinel must not survive, even if the SQL contains sentinel-looking text.""" sql = 'SELECT "a.b", \'\\x00 0 \\x00\' AS lit' got = substitute_quoted(sql, {"a.b": "z"}, quote=_dq) assert "\x00" not in got.replace("\\x00", "") -# --------------------------------------------------------------------------- -# Dialect wiring -# --------------------------------------------------------------------------- - - -# Conservative universal byte budgets. NOT an exact model of each backend's -# per-identifier-class rules: MySQL's 64-char *identifier* limit is used rather -# than its 256-char *column-alias* limit, and Oracle assumes 12.2+ (128, not the -# pre-12.2 30). Bytes are conservative for char-counting backends. +# Conservative universal byte budgets, not exact per-identifier-class rules: MySQL uses its +# 64-char identifier limit (not the 256-char column-alias one) and Oracle assumes 12.2+ (128). EXPECTED_LIMITS = { "postgres": 63, "mysql": 64, @@ -370,16 +296,14 @@ def test_configured_limit(self, name: str, expected: int | None) -> None: assert get_dialect(name).max_identifier_bytes == expected def test_base_default_is_conservative(self) -> None: - """A future dialect that forgets to set the field must inherit the - TIGHTEST limit, not an unbounded one — over-shortening is safe.""" + """A dialect that forgets to set the field inherits the tightest limit, not unbounded.""" from slayer.sql.dialects.base import SqlDialect assert SqlDialect().max_identifier_bytes == 63 @pytest.mark.parametrize("name", sorted(EXPECTED_LIMITS)) def test_emit_alias_identity_under_limit(self, name: str) -> None: - """Only BigQuery/T-SQL transform a short alias (dot-mangling); every - other dialect must leave it byte-identical.""" + """Only BigQuery/T-SQL dot-mangle a short alias; every other dialect leaves it byte-identical.""" got = get_dialect(name).emit_alias("orders.revenue_sum") if name in ("bigquery", "tsql"): assert got == encode_alias("orders.revenue_sum") @@ -388,8 +312,7 @@ def test_emit_alias_identity_under_limit(self, name: str) -> None: @pytest.mark.parametrize("name", sorted(EXPECTED_LIMITS)) def test_fit_alias_identity_under_limit(self, name: str) -> None: - """``fit_alias`` is the LENGTH-ONLY half — identity on every dialect - for a short alias, which is what makes the write pass a no-op.""" + """``fit_alias`` is length-only, so a short alias is identity on every dialect.""" assert get_dialect(name).fit_alias("orders.revenue_sum") == "orders.revenue_sum" @pytest.mark.parametrize("name", ["sqlite", "clickhouse", "trino", "presto", "databricks", "spark"]) @@ -402,8 +325,7 @@ def test_postgres_shortens_over_limit(self) -> None: assert _nbytes(got) <= 63 def test_bigquery_emit_alias_is_mangled_and_fitted(self) -> None: - """BigQuery's ``emit_alias`` must be the FINAL identifier reaching the - SQL: length-fitted first, then dot-mangled.""" + """``emit_alias`` is the final identifier: length-fitted first, then dot-mangled.""" long_dotted = ".".join(["Sandbox" * 6] * 8) # way over 300 bq = get_dialect("bigquery") got = bq.emit_alias(long_dotted) @@ -420,9 +342,7 @@ def test_tsql_emit_alias_is_mangled_and_fitted(self) -> None: assert got == encode_alias(tsql.fit_alias(long_dotted)) -# --------------------------------------------------------------------------- # alias_rewrite_map — the collision guard -# --------------------------------------------------------------------------- class TestAliasRewriteMap: @@ -442,14 +362,11 @@ def test_unbounded_dialect_map_is_empty(self) -> None: assert get_dialect("sqlite").alias_rewrite_map([LONG_NAME, LONG_EMAIL]) == {} def test_duplicate_canonical_aliases_are_not_a_collision(self) -> None: - """The same alias listed twice is one name, not two.""" pg = get_dialect("postgres") assert pg.alias_rewrite_map([LONG_EMAIL, LONG_EMAIL]) == pg.alias_rewrite_map([LONG_EMAIL]) def test_keys_and_values_are_disjoint(self) -> None: - """The invariant that makes the write pass safe: every key is OVER the - limit and every value is WITHIN it, so no substitution can produce - another key. (The two-phase pass defends this if it ever weakens.)""" + """Every key is over the limit and every value within it, so no substitution can produce a key.""" pg = get_dialect("postgres") mapping = pg.alias_rewrite_map([LONG_NAME, LONG_EMAIL, "orders.status"]) assert mapping @@ -458,13 +375,7 @@ def test_keys_and_values_are_disjoint(self) -> None: assert _nbytes(key) > 63 >= _nbytes(value) def test_digest_collision_raises(self, monkeypatch: pytest.MonkeyPatch) -> None: - """Force two distinct over-limit aliases onto one emitted name. - - Note the pair differs only in the MIDDLE — head and tail both survive - fitting, so the repro pair (which differs in its final segment) cannot - be used here: it stays distinct even with a constant digest, which is - exactly the readability property the shape is chosen for. - """ + """A constant digest forces two mid-differing over-limit aliases onto one emitted name.""" import slayer.sql.dialects._identifier_fit as fitmod monkeypatch.setattr(fitmod, "_digest", lambda name: "deadbeef") @@ -475,9 +386,7 @@ def test_digest_collision_raises(self, monkeypatch: pytest.MonkeyPatch) -> None: assert TWIN_B in str(exc.value) def test_shortened_form_equal_to_an_existing_short_alias_raises(self) -> None: - """The guard must consider IDENTITY entries too. A short alias whose - spelling equals another alias's fitted form is a duplicate output name - that hash width alone cannot prevent.""" + """The guard covers identity entries: a short alias equal to another's fitted form is a duplicate.""" pg = get_dialect("postgres") collider = pg.fit_alias(LONG_EMAIL) # within 63 bytes, so an identity entry assert pg.fit_alias(collider) == collider @@ -501,15 +410,12 @@ def test_is_a_slayer_error_and_value_error(self) -> None: assert issubclass(IdentifierCollisionError, ValueError) -# --------------------------------------------------------------------------- # decode_result_keys — many-to-one must not silently overwrite -# --------------------------------------------------------------------------- class TestDecodeCollision: def test_two_keys_decoding_to_one_canonical_raises(self) -> None: - """A row carrying both the fitted form of an alias AND that alias's own - canonical spelling would silently lose one value on ``dict`` rebuild.""" + """A row carrying both an alias's fitted form and its canonical spelling must not lose a value.""" pg = get_dialect("postgres") rows = [{pg.fit_alias(LONG_EMAIL): 1, LONG_EMAIL: 2}] with pytest.raises(IdentifierCollisionError): @@ -517,11 +423,7 @@ def test_two_keys_decoding_to_one_canonical_raises(self) -> None: @pytest.mark.parametrize("dialect", ["bigquery", "tsql"]) def test_mangle_fallback_collision_raises(self, dialect: str) -> None: - """The dot-mangling dialects decode unmapped keys through - ``decode_alias``. That fallback must run INSIDE the duplicate check — - pre-decoding into a dict first would let ``orders___status`` and - ``orders.status`` collapse onto one key with one value silently - dropped, before any collision could be observed.""" + """The ``decode_alias`` fallback runs inside the duplicate check so ``orders___status`` and ``orders.status`` collide.""" d = get_dialect(dialect) rows = [{"orders___status": 1, "orders.status": 2}] with pytest.raises(IdentifierCollisionError): @@ -529,7 +431,6 @@ def test_mangle_fallback_collision_raises(self, dialect: str) -> None: @pytest.mark.parametrize("dialect", ["bigquery", "tsql"]) def test_mangle_fallback_preserves_every_value(self, dialect: str) -> None: - """No key is dropped when nothing collides.""" d = get_dialect(dialect) got = d.decode_result_keys([{"a___b": 1, "c___d": 2}], aliases=[]) assert got == [{"a.b": 1, "c.d": 2}] diff --git a/tests/integration/test_dev1756_identifier_length_pg.py b/tests/integration/test_dev1756_identifier_length_pg.py index f325b800..dcd3fe94 100644 --- a/tests/integration/test_dev1756_identifier_length_pg.py +++ b/tests/integration/test_dev1756_identifier_length_pg.py @@ -1,19 +1,7 @@ -"""DEV-1756 on a REAL Postgres server. - -The whole point of this issue is that byte-level emission tests pass while the -server rejects (or worse, silently mis-answers) the query. Postgres caps -identifiers at 63 bytes and truncates past it with only a NOTICE, so these -failures are invisible to any test that only inspects generated SQL. - -Two distinct failure modes are covered: - -* **Collapse** — two sibling aliases share a 63-byte prefix. With the DEV-1444 - outer wrap in play the re-projection is ambiguous (``AmbiguousColumnError``); - without it the two columns collapse into one in the result row. -* **Silent loss** — a SINGLE over-limit alias with no sibling. Postgres accepts - the query and returns the row keyed by the truncated name, so the engine's - canonical-alias lookup misses and a column silently disappears. No error is - raised anywhere, which makes this the more dangerous of the two. +"""DEV-1756 against a live Postgres, which truncates identifiers past 63 bytes with +only a NOTICE, so emission tests pass while the server mis-answers. Covers two modes: +sibling aliases collapsing at a shared 63-byte prefix, and a lone over-limit alias +vanishing with no error (the more dangerous case). """ import re @@ -64,8 +52,7 @@ def _drop_db(postgresql_proc, db_name): @pytest.fixture(scope="module") def _chain_storage(postgresql_proc, tmp_path_factory): - """3-hop join chain with model names long enough that the projection - aliases cross Postgres' 63-byte limit.""" + """3-hop join chain whose projection aliases cross Postgres' 63-byte limit.""" conn, db_name = _create_db(postgresql_proc) try: cur = conn.cursor() @@ -153,8 +140,7 @@ def chain_env(_chain_storage): @pytest.mark.integration class TestPostgresIdentifierLength: async def test_server_truncates_at_63_bytes(self, chain_env) -> None: - """Pin the premise against the actual server, so the rest of this file - cannot pass for the wrong reason if NAMEDATALEN ever differs.""" + """Pin the 63-byte truncation premise against the live server.""" client = chain_env._get_client( await chain_env._resolve_datasource( model=await chain_env.storage.get_model("SandboxInvoiceV2", data_source=DS), @@ -166,9 +152,7 @@ async def test_server_truncates_at_63_bytes(self, chain_env) -> None: assert len(list(rows[0])[0].encode()) == 63 async def test_collapse_with_outer_wrap(self, chain_env) -> None: - """The exact reported failure: two 73/74-byte siblings + an ORDER BY - hoist that forces the DEV-1444 outer wrap. Raised AmbiguousColumnError - before the fix.""" + """Two over-limit siblings + ORDER BY outer wrap; raised AmbiguousColumnError before the fix.""" query = SlayerQuery( source_model="SandboxInvoiceV2", dimensions=[ @@ -181,8 +165,7 @@ async def test_collapse_with_outer_wrap(self, chain_env) -> None: limit=10, ) result = await chain_env.execute(query=query) - # Exact rows: a fitted alias that resolved to the WRONG column would - # still be "present with distinct values", so pin the actual pairing. + # Pin the exact pairing; a wrong-column alias would still be "present with distinct values". got = { (row[LONG_NAME], row[LONG_EMAIL], row["SandboxInvoiceV2.status"]): (float(row["SandboxInvoiceV2.totalAmount_sum"]), row["SandboxInvoiceV2._count"]) @@ -194,8 +177,7 @@ async def test_collapse_with_outer_wrap(self, chain_env) -> None: } async def test_collapse_without_outer_wrap(self, chain_env) -> None: - """No ORDER BY, so no outer wrap and no server-side error — the two - columns would silently collapse into one result key.""" + """No outer wrap: the two siblings would silently collapse into one result key.""" query = SlayerQuery( source_model="SandboxInvoiceV2", dimensions=[ @@ -212,9 +194,7 @@ async def test_collapse_without_outer_wrap(self, chain_env) -> None: assert emails == {"ann@example.io", "bob@example.io"} async def test_silent_loss_single_over_limit_alias(self, chain_env) -> None: - """The dangerous mode: ONE over-limit alias, no sibling to collide - with. Postgres accepts the query and keys the row by the truncated - name, so the column silently vanishes from the response.""" + """One over-limit alias, no sibling: Postgres keys the row by the truncated name and it vanishes.""" query = SlayerQuery( source_model="SandboxInvoiceV2", dimensions=[ColumnRef(name=f"{DEEP}.email")], @@ -229,8 +209,7 @@ async def test_silent_loss_single_over_limit_alias(self, chain_env) -> None: assert row[LONG_EMAIL] in {"ann@example.io", "bob@example.io"} async def test_values_are_correct_not_merely_present(self, chain_env) -> None: - """A fitted alias that pointed at the wrong column would still be - 'present'. Pin the actual aggregate.""" + """Pin the aggregate; a wrong-column fitted alias would still be present.""" query = SlayerQuery( source_model="SandboxInvoiceV2", dimensions=[ColumnRef(name=f"{DEEP}.email")], @@ -241,8 +220,7 @@ async def test_values_are_correct_not_merely_present(self, chain_env) -> None: assert totals == {"ann@example.io": 150.0, "bob@example.io": 200.0} async def test_response_columns_are_canonical(self, chain_env) -> None: - """Consumers must never see the shortened form — that is what the - read-side decode exists for.""" + """Response exposes canonical keys while the emitted SQL carries the fitted form.""" query = SlayerQuery( source_model="SandboxInvoiceV2", dimensions=[ColumnRef(name=f"{DEEP}.name"), ColumnRef(name=f"{DEEP}.email")], @@ -250,12 +228,10 @@ async def test_response_columns_are_canonical(self, chain_env) -> None: ) result = await chain_env.execute(query=query) assert set(result.data[0]) >= {LONG_NAME, LONG_EMAIL} - # ...while the SQL that actually ran carries the fitted form. assert LONG_EMAIL not in result.sql async def test_cached_execution_round_trip(self, chain_env) -> None: - """The cache stores the DECODED response and re-keys on the fitted - SQL; a hit must return canonical keys too.""" + """A cache hit returns canonical keys too, proving decode runs before storage.""" query = SlayerQuery( source_model="SandboxInvoiceV2", dimensions=[ColumnRef(name=f"{DEEP}.email")], @@ -263,34 +239,27 @@ async def test_cached_execution_round_trip(self, chain_env) -> None: ) first = await chain_env.execute(query=query, cache=True) second = await chain_env.execute(query=query, cache=True) - # Canonical on BOTH the fresh result and the cache hit — proving the - # decode runs before storage, not only on the way out. assert LONG_EMAIL in first.data[0] assert LONG_EMAIL in second.data[0] assert first.data == second.data async def test_deep_cross_model_measure_executes(self, chain_env) -> None: - """Surface 3: the `_cm_` CTE name is `_cm_` + the dotted alias, which - also crosses 63 bytes on this chain.""" + """Surface 3: the `_cm_` cross-model CTE name also crosses 63 bytes on this chain.""" query = SlayerQuery( source_model="SandboxInvoiceV2", dimensions=[ColumnRef(name="status")], measures=[{"formula": f"{DEEP}.lifetimeValue:sum"}], ) result = await chain_env.execute(query=query) - # The construct under test must actually be present... assert "_cm_" in result.sql, "no cross-model CTE generated; test is vacuous" for name in re.findall(r"\b_cm_\w+", result.sql): assert len(name.encode()) <= 63, f"{name!r} exceeds the Postgres limit" - # ...and the value must be right, not merely non-empty. alias = f"SandboxInvoiceV2.{DEEP}.lifetimeValue_sum" assert len(result.data) == 1 assert float(result.data[0][alias]) == 30.0 async def test_nested_query_backed_model_executes(self, chain_env) -> None: - """Surface 4: the virtual-model short names are emitted as output - column aliases and referenced by the outer stage. Mixed-case shorts - additionally exercise the case-folding regression.""" + """Surface 4: virtual-model short names as output aliases; mixed-case exercises case-folding.""" stage1 = SlayerQuery( name="stage1", source_model="SandboxInvoiceV2", diff --git a/tests/test_dev1756_identifier_length.py b/tests/test_dev1756_identifier_length.py index 124232f6..7625be33 100644 --- a/tests/test_dev1756_identifier_length.py +++ b/tests/test_dev1756_identifier_length.py @@ -1,28 +1,8 @@ -"""DEV-1756: SLayer must bound generated identifiers to the dialect's limit. +"""SLayer must bound generated identifiers to the dialect's limit. -Postgres caps identifiers at 63 BYTES and SILENTLY truncates past it. SLayer's -projection aliases (``..``) cross that on a 3-hop -join, so two siblings collapse onto one effective output name: - - SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name 73 B - SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email 74 B - -> both truncate to ...SandboxCustomer.SandboxCon 63 B - -With the DEV-1444 outer wrap in play that raises ``AmbiguousColumnError``; -without it, the two columns silently collapse in the result row. - -Three surfaces are fixed here: - -1. Projection aliases -- QUOTED; inner SELECT, outer wrap, ORDER BY. -3. CTE names -- UNQUOTED; ``_cte_name_from_alias``. -4. Virtual-model shorts -- ``_query_as_model``'s ``_alias_to_short``. - -Surface 2 (join-path TABLE aliases such as -``SandboxSubscription__SandboxCustomer__SandboxConsumer``) is DEFERRED to -DEV-1743 and is deliberately NOT asserted on here — see ``_inscope_identifiers``. - -The primitive itself is unit-tested in ``tests/dialects/test_identifier_fit.py``; -live execution against a real server is in the Postgres integration suite. +Postgres caps identifiers at 63 bytes and silently truncates past it, so long +projection aliases collide. Fixed here: projection aliases (quoted), CTE names +(unquoted), virtual-model shorts. Table aliases are surface 2, deferred to DEV-1743. """ from __future__ import annotations @@ -48,17 +28,12 @@ LONG_NAME = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name" LONG_EMAIL = "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email" -# Two over-limit names differing ONLY in the middle — head and tail survive -# fitting identically, which is what makes a forced digest collision possible. +# Two over-limit names differing only in the middle: forces a digest collision. TWIN_A = "SandboxAlpha." * 3 + "111" + ".SandboxOmega" * 3 TWIN_B = "SandboxAlpha." * 3 + "222" + ".SandboxOmega" * 3 -# --------------------------------------------------------------------------- -# Pre-change golden SQL — captured from the generator BEFORE this feature -# existed. These pin the "no churn for the common case" guarantee far more -# strongly than an idempotence check could. -# --------------------------------------------------------------------------- +# Pre-change golden SQL — pins "no churn for the common case". GOLDEN_SHORT_QUERY = { "postgres": ( @@ -88,8 +63,7 @@ ), } -# The full repro (long aliases + outer wrap) on an UNBOUNDED dialect: nothing -# may change, byte for byte. +# Full repro on an unbounded dialect: nothing may change, byte for byte. GOLDEN_SQLITE_REPRO_ORDER = ( 'SELECT\n "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.name",\n' ' "SandboxInvoiceV2.SandboxSubscription.SandboxCustomer.SandboxConsumer.email",\n' @@ -117,9 +91,7 @@ ) -# --------------------------------------------------------------------------- -# Fixtures — the reported 3-hop chain, with realistic model-name lengths -# --------------------------------------------------------------------------- +# Fixtures — the reported 3-hop chain def _chain_models( @@ -134,8 +106,7 @@ def _chain_models( Column(name="lifetimeValue", sql="lifetime_value", type=DataType.DOUBLE), ] if case_colliding_columns: - # Differs from ``email`` only by case: the derived virtual-model shorts - # are emitted into a namespace Postgres case-folds. + # Differs from ``email`` only by case; Postgres case-folds the namespace. consumer_columns.append(Column(name="Email", sql="email", type=DataType.TEXT)) return [ SlayerModel( @@ -146,9 +117,7 @@ def _chain_models( Column(name="totalAmount", sql="total_amount", type=DataType.DOUBLE), Column(name="subscription_id", sql="subscription_id", type=DataType.INT), *([ - # A root column named exactly what a deep join path flattens - # to. Column names permit ``__`` (the query-backed carve-out), - # so this is reachable, not contrived. + # Root column named what a deep join path flattens to; names permit ``__``. Column(name=decoy_root_column, sql="decoy", type=DataType.TEXT), ] if decoy_root_column else []), ], @@ -222,14 +191,8 @@ def _repro_query(*, with_order: bool = False) -> SlayerQuery: ) -# --------------------------------------------------------------------------- -# Identifier-inspection helpers -# -# Scope note: these deliberately inspect ONLY the namespaces this issue fixes — -# output-column aliases, ORDER BY references and CTE names. Join-path TABLE -# aliases (``exp.TableAlias``) are surface 2, deferred to DEV-1743, and are -# excluded so a future long join chain fails THERE rather than confusingly here. -# --------------------------------------------------------------------------- +# Identifier-inspection helpers — only the namespaces this issue owns (aliases, +# ORDER BY refs, CTE names). Join-path table aliases are surface 2 (DEV-1743). def _nbytes(s: str) -> int: @@ -237,8 +200,7 @@ def _nbytes(s: str) -> int: def _pg_effective(name: str, *, quoted: bool) -> str: - """What Postgres actually resolves an identifier to: truncate to 63 bytes, - and additionally case-fold when it was written unquoted.""" + """What Postgres resolves an identifier to: truncate to 63 bytes, case-fold if unquoted.""" clipped = name.encode("utf-8")[:63].decode("utf-8", "ignore") return clipped if quoted else clipped.lower() @@ -275,8 +237,7 @@ def _cte_names(tree: exp.Expression) -> list[tuple[str, bool]]: def _cte_table_refs(tree: exp.Expression) -> set[str]: - """Names of every table reference, so a CTE reference can be matched - against its definition EXACTLY rather than by substring count.""" + """Names of every table reference, to match a CTE reference to its definition exactly.""" return { t.this.this for t in tree.find_all(exp.Table) @@ -301,9 +262,7 @@ def _assert_within_limit(sql: str, limit: int, dialect: str = "postgres") -> Non def _assert_no_namespace_collision(sql: str, dialect: str = "postgres") -> None: - """Within each namespace, identifiers must stay distinct AFTER the backend's - normalization (truncate, plus case-fold when unquoted) — not merely be - short enough.""" + """Identifiers in each namespace stay distinct after the backend normalizes them.""" tree = sqlglot.parse_one(sql, dialect=dialect) namespaces: list[tuple[str, list[tuple[str, bool]]]] = [ (f"select#{i}", _projection_aliases(select)) @@ -322,9 +281,7 @@ def _assert_no_namespace_collision(sql: str, dialect: str = "postgres") -> None: def _assert_order_by_refs_resolve(sql: str, dialect: str = "postgres") -> None: - """Every quoted ORDER BY reference must name a projection alias that - actually exists somewhere in the statement. This is the pairing check that - catches an ORDER BY left pointing at an unfitted alias.""" + """Every quoted ORDER BY reference must name a projection alias that exists.""" tree = sqlglot.parse_one(sql, dialect=dialect) projected = {n for select in tree.find_all(exp.Select) for n, _ in _projection_aliases(select)} for select in tree.find_all(exp.Select): @@ -340,9 +297,7 @@ async def _sql(engine, model, query, *, dialect: str = "postgres", mode: str = " return SQLGenerator(dialect=dialect).generate(enriched=enriched, render_mode=mode) -# =========================================================================== -# 1. The premise — without the fix these aliases really do collide -# =========================================================================== +# The premise: without the fix these aliases really do collide class TestPremise: @@ -359,11 +314,6 @@ def test_the_two_aliases_share_a_63_byte_prefix(self) -> None: assert LONG_NAME.encode()[:63] == LONG_EMAIL.encode()[:63] -# =========================================================================== -# 2. Surface 1 — projection aliases -# =========================================================================== - - class TestProjectionAliases: async def test_repro_emits_no_over_limit_identifier(self, chain) -> None: engine, model = chain @@ -379,8 +329,7 @@ async def test_repro_still_parses(self, chain) -> None: assert len(sqlglot.parse(sql, dialect="postgres")) == 1 async def test_outer_wrap_uses_the_same_token_everywhere(self, chain) -> None: - """The reported failure: the inner ``AS ``, the outer wrap's - projection and the ORDER BY must all carry the IDENTICAL identifier.""" + """Inner ``AS``, outer-wrap projection and ORDER BY carry the identical token.""" engine, model = chain sql = await _sql(engine, model, _repro_query(with_order=True)) assert ") AS _outer" in sql, "outer wrap did not fire; test is vacuous" @@ -396,8 +345,7 @@ async def test_outer_wrap_uses_the_same_token_everywhere(self, chain) -> None: _assert_order_by_refs_resolve(sql) async def test_canonical_alias_does_not_survive_anywhere(self, chain) -> None: - """Pairing check: if ANY occurrence were missed, the definition and its - references would disagree. Stronger than a max-length assertion.""" + """Pairing check: no canonical alias survives — stronger than a max-length assertion.""" engine, model = chain sql = await _sql(engine, model, _repro_query(with_order=True)) assert LONG_NAME not in sql @@ -405,14 +353,12 @@ async def test_canonical_alias_does_not_survive_anywhere(self, chain) -> None: @pytest.mark.parametrize("dialect", sorted(GOLDEN_SHORT_QUERY)) async def test_under_limit_query_is_byte_identical(self, chain, dialect: str) -> None: - """No churn for the 99% case, pinned against SQL captured BEFORE this - feature existed.""" + """No churn for the common case, pinned against pre-feature SQL.""" engine, model = chain assert await _sql(engine, model, SHORT_QUERY, dialect=dialect) == GOLDEN_SHORT_QUERY[dialect] async def test_unbounded_dialect_output_is_byte_identical(self, chain) -> None: - """SQLite has no limit, so even the full repro — long aliases, outer - wrap, ORDER BY — must be untouched byte for byte.""" + """SQLite has no limit, so even the full repro is untouched byte for byte.""" engine, model = chain sql = await _sql(engine, model, _repro_query(with_order=True), dialect="sqlite") assert sql == GOLDEN_SQLITE_REPRO_ORDER @@ -421,19 +367,16 @@ async def test_unbounded_dialect_output_is_byte_identical(self, chain) -> None: async def test_other_unbounded_dialects_keep_the_long_alias(self, chain, dialect: str) -> None: engine, model = chain sql = await _sql(engine, model, _repro_query(with_order=True), dialect=dialect) - # Inner AS + outer projection: both untouched. - assert sql.count(LONG_EMAIL) == 2 + assert sql.count(LONG_EMAIL) == 2 # inner AS + outer projection, both untouched async def test_wrapped_render_mode_also_fitted(self, chain) -> None: - """``render_mode='wrapped'`` feeds ``_query_as_model``; its inner - aliases are just as subject to truncation.""" + """``render_mode='wrapped'`` inner aliases are subject to truncation too.""" engine, model = chain _assert_within_limit(await _sql(engine, model, _repro_query(), mode="wrapped"), 63) class TestManglingDialects: - """BigQuery / T-SQL already mangle dotted aliases; length-fitting must - compose with that, not fight it.""" + """BigQuery / T-SQL mangle dotted aliases; length-fitting must compose with that.""" @pytest.mark.parametrize("dialect,limit", [("bigquery", 300), ("tsql", 128)]) async def test_long_alias_is_mangled_and_fitted(self, chain, dialect: str, limit: int) -> None: @@ -443,8 +386,7 @@ async def test_long_alias_is_mangled_and_fitted(self, chain, dialect: str, limit assert LONG_EMAIL not in sql async def test_emit_alias_matches_what_is_in_the_sql(self, chain) -> None: - """``emit_alias`` is what the decode map is built from, so it must be - exactly the token the SQL carries.""" + """``emit_alias`` builds the decode map, so it must be the token the SQL carries.""" engine, model = chain for dialect in ("postgres", "bigquery", "tsql", "mysql"): sql = await _sql(engine, model, _repro_query(), dialect=dialect) @@ -452,9 +394,7 @@ async def test_emit_alias_matches_what_is_in_the_sql(self, chain) -> None: assert get_dialect(dialect).emit_alias(LONG_EMAIL) in names, dialect async def test_mangling_is_not_applied_twice(self, chain) -> None: - """The base length pass runs BEFORE the dot-mangle regex. If it emitted - an already-mangled form the regex would double-encode ``___`` to - ``______``.""" + """The length pass runs before the dot-mangle regex; it must not double-encode.""" engine, model = chain sql = await _sql(engine, model, _repro_query(), dialect="bigquery") fitted = get_dialect("bigquery").fit_alias(LONG_EMAIL) @@ -462,11 +402,6 @@ async def test_mangling_is_not_applied_twice(self, chain) -> None: assert encode_alias(encode_alias(fitted)) not in sql -# =========================================================================== -# 3. Read side -# =========================================================================== - - class TestDecodeResultKeys: def test_shortened_keys_restored_to_canonical(self) -> None: pg = get_dialect("postgres") @@ -490,8 +425,7 @@ def test_empty_rows(self) -> None: assert get_dialect("postgres").decode_result_keys([], aliases=[LONG_EMAIL]) == [] def test_hidden_alias_is_decoded_too(self) -> None: - """Hidden ORDER-BY hoists are projected in the inner SELECT, so a row - can legitimately carry one; it must decode like any other.""" + """A hidden ORDER-BY hoist can appear in a row and must decode like any other.""" pg = get_dialect("postgres") hidden = LONG_EMAIL.replace(".email", ".totalAmount_avg") rows = [{pg.emit_alias(hidden): 1.0}] @@ -504,16 +438,12 @@ def test_bigquery_reverses_both_manglings(self) -> None: assert bq.decode_result_keys(rows, aliases=[long_dotted]) == [{long_dotted: 1}] def test_bigquery_falls_back_to_dot_decode_outside_the_map(self) -> None: - """Keys not in the length map must still get today's ``___`` -> ``.`` - treatment, or short-alias BigQuery results would regress.""" bq = get_dialect("bigquery") assert bq.decode_result_keys([{"orders___status": 1}], aliases=[]) == [{"orders.status": 1}] class TestDecodeWiring: - """The decode must be handed the FULL alias set, hidden entries included — - an implementation that used only the public aliases would still pass the - end-to-end repro.""" + """Decode must receive the full alias set, hidden entries included.""" async def test_run_and_build_passes_all_projection_aliases( self, chain, monkeypatch: pytest.MonkeyPatch, @@ -544,11 +474,6 @@ async def execute(self, sql): assert seen["aliases"] == expected -# =========================================================================== -# 4. The alias set -# =========================================================================== - - class TestAllProjectionAliases: async def test_includes_public_aliases(self, chain) -> None: engine, model = chain @@ -558,9 +483,7 @@ async def test_includes_public_aliases(self, chain) -> None: assert a in every async def test_includes_hidden_order_by_hoist(self, chain) -> None: - """``totalAmount:avg`` is projected in the inner SELECT purely to - satisfy ORDER BY. It truncates like any other alias, so the pass must - see it.""" + """``totalAmount:avg`` is projected only for ORDER BY; the pass must still see it.""" engine, model = chain enriched = await engine._enrich(query=_repro_query(with_order=True), model=model) public = public_projection_aliases(enriched) @@ -568,8 +491,7 @@ async def test_includes_hidden_order_by_hoist(self, chain) -> None: assert any("totalAmount_avg" in a for a in hidden), hidden async def test_is_stable_across_calls(self, chain) -> None: - """Order must not depend on set iteration — the rewrite map is derived - from this list.""" + """Order must be stable — the rewrite map is derived from this list.""" engine, model = chain enriched = await engine._enrich(query=_repro_query(), model=model) first = all_projection_aliases(enriched) @@ -580,9 +502,7 @@ async def test_is_stable_across_calls(self, chain) -> None: assert all_projection_aliases(enriched) == first -# =========================================================================== -# 5. Surface 3 — CTE names -# =========================================================================== +# CTE names (unquoted namespace) def _deep_cross_model_query(*, two_measures: bool = False) -> SlayerQuery: @@ -604,9 +524,7 @@ async def test_cross_model_cte_name_within_limit(self, chain) -> None: _assert_within_limit(sql, 63) async def test_cte_definition_and_references_agree(self, chain) -> None: - """A CTE name is emitted UNQUOTED, so a truncated definition and an - untruncated reference would silently disagree. Compare parsed - identifiers, not substring counts.""" + """A CTE name is unquoted; a truncated definition and untruncated reference must agree.""" engine, model = chain sql = await _sql(engine, model, _deep_cross_model_query()) tree = sqlglot.parse_one(sql, dialect="postgres") @@ -618,8 +536,7 @@ async def test_cte_definition_and_references_agree(self, chain) -> None: assert name in referenced, f"CTE {name!r} defined but never referenced\n{sql}" async def test_two_deep_cross_model_ctes_stay_distinct(self, chain) -> None: - """Two deep cross-model measures produce two over-limit CTE names; both - must fit AND stay distinct after the server's truncation.""" + """Two over-limit cross-model CTE names must fit and stay distinct after truncation.""" engine, model = chain sql = await _sql(engine, model, _deep_cross_model_query(two_measures=True)) tree = sqlglot.parse_one(sql, dialect="postgres") @@ -630,16 +547,7 @@ async def test_two_deep_cross_model_ctes_stay_distinct(self, chain) -> None: _assert_within_limit(sql, 63) def test_cte_namespace_collision_raises(self, monkeypatch: pytest.MonkeyPatch) -> None: - """Forced digest collision in the CTE namespace must raise rather than - emit two identically-named CTEs — which would silently make one of them - reference the other's rows. - - Driven through the allocator directly: CTE names derive from measure - aliases, and two aliases reachable from one query always differ in - their final segment, which fitting preserves. So a colliding PAIR is - not constructible from a natural query — the allocator is the thing - that has to hold the invariant. - """ + """A forced digest collision in the CTE namespace must raise, not emit duplicate names.""" import slayer.sql.dialects._identifier_fit as fitmod gen = SQLGenerator(dialect="postgres") @@ -651,23 +559,14 @@ def test_cte_namespace_collision_raises(self, monkeypatch: pytest.MonkeyPatch) - assert "CTE name" in str(exc.value) def test_cte_allocator_is_idempotent_for_the_same_owner(self) -> None: - """Several code paths re-derive a CTE's name in order to reference it; - that must not read as a collision.""" + """Re-deriving a CTE's name for the same owner must not read as a collision.""" gen = SQLGenerator(dialect="postgres") first = gen._cte_name("_cm_", TWIN_A) second = gen._cte_name("_cm_", TWIN_A) # hits the memo, must not raise assert second == first async def test_cte_allocator_resets_per_statement(self, chain) -> None: - """One generator instance generates many statements; allocation is - per-statement, so an owner allocated before ``generate()`` must not - still hold its name afterwards. - - The reset has to be exercised THROUGH ``generate()``, and with a - DIFFERENT owner: re-generating the same statement proves nothing, - because ``_cte_name`` is idempotent for one owner and would pass even - if the dict were never cleared. - """ + """CTE allocation is per-statement; ``generate()`` clears an owner allocated before it.""" engine, _ = chain prepared = await engine._prepare_pipeline( query=_repro_query(), named_queries={}, runtime_kwarg={}, @@ -682,11 +581,7 @@ async def test_cte_allocator_resets_per_statement(self, chain) -> None: ) def test_cte_allocator_detects_case_folded_collision(self) -> None: - """CTE names are emitted UNQUOTED, so the server folds them: on - Postgres ``_wm_Foo`` and ``_wm_foo`` are the SAME identifier, and - allocating both would silently point one reference at the other's CTE. - Comparing the literal spellings misses it. - """ + """CTE names are unquoted and case-folded, so ``_wm_Foo`` and ``_wm_foo`` collide.""" gen = SQLGenerator(dialect="postgres") first = gen._cte_name(prefix="_wm_", alias="Foo") with pytest.raises(IdentifierCollisionError) as exc: @@ -704,16 +599,12 @@ def test_cte_name_helper_is_pure_and_bounded(self) -> None: assert _nbytes(a) <= 63 def test_cte_name_helper_counts_the_prefix(self) -> None: - """The budget covers the WHOLE emitted name, prefix included — a - ``cp_value_12_`` prefix eats 12 of the 63 bytes.""" from slayer.sql.generator import _cte_name_from_alias assert _nbytes(_cte_name_from_alias("cp_value_12_", "a" * 60, limit=63)) <= 63 def test_cte_name_helper_unbounded(self) -> None: - """``limit=None`` keeps today's behaviour exactly. The alias here is - already flat, so sanitization is a no-op and the result is a plain - concatenation.""" + """``limit=None`` keeps today's behaviour exactly.""" from slayer.sql.generator import _cte_name_from_alias assert _cte_name_from_alias("_cm_", "a" * 200, limit=None) == "_cm_" + "a" * 200 @@ -739,11 +630,7 @@ def test_cte_name_is_a_legal_unquoted_identifier(self) -> None: assert re.match(r"^[A-Za-z_][A-Za-z0-9_]*$", got), got async def test_self_join_transform_cte_names_are_fitted(self, tmp_path) -> None: - """``shifted_`` / ``sjoin_`` are CTE names built from a - USER-supplied transform name, so a long one busts the budget just like - an alias-derived CTE. They used to be f-string-built and so escaped - both the fitting and the collision check. - """ + """``shifted_``/``sjoin_`` CTE names come from a user transform name and must fit too.""" long_transform_name = "revenue_" + "x" * 70 # 78 chars, way over 63 storage = YAMLStorage(base_dir=str(tmp_path)) await storage.save_datasource(DatasourceConfig( @@ -775,16 +662,11 @@ async def test_self_join_transform_cte_names_are_fitted(self, tmp_path) -> None: assert "shifted_" in sql, f"fixture must emit a self-join CTE\n{sql}" for name, _ in _cte_names(sqlglot.parse_one(sql, dialect="postgres")): assert _nbytes(name) <= 63, f"CTE {name!r} is {_nbytes(name)} bytes\n{sql}" - # The unfitted forms must not survive anywhere — definition or reference. + # The unfitted forms must not survive — definition or reference. assert f"shifted_{long_transform_name}" not in sql assert f"sjoin_{long_transform_name}" not in sql -# =========================================================================== -# 6. Surface 4 — _query_as_model short names -# =========================================================================== - - def _wrapper_select(vm_sql: str) -> exp.Select: """The outermost SELECT of a virtual model's SQL — the rename wrapper.""" return next(iter(sqlglot.parse_one(vm_sql, dialect="postgres").find_all(exp.Select))) @@ -811,16 +693,7 @@ async def test_long_siblings_get_distinct_shorts(self, chain) -> None: async def test_short_alias_quoting_matches_the_downstream_reference( self, chain, ) -> None: - """The wrapper's ``AS `` must be quoted exactly when the - downstream ``Column(sql=short)`` reference is. - - Emitted bare, a MIXED-CASE short is case-folded by Postgres while the - outer stage references it quoted (``_quote_mixed_case_identifiers``) - -> ``UndefinedColumnError``. But quoting unconditionally breaks the - mirror image on UPPER-folding backends: a case-sensitive ``"status"`` - would be defined while the bare reference resolves as ``STATUS``. The - contract is agreement, not quoting. - """ + """The wrapper's ``AS `` is quoted exactly when the downstream reference is.""" engine, _ = chain vm = await engine._query_as_model(inner_query=_repro_query()) gen = SQLGenerator(dialect="postgres") @@ -875,18 +748,7 @@ async def test_mixed_case_short_is_quoted(self, chain) -> None: def test_short_spelling_matches_the_reference_on_every_dialect( self, dialect: str, short: str, ) -> None: - """``AS `` must be spelled exactly as a downstream - ``Column(sql=short)`` reference to it will be, on every dialect. - - Two independent axes, both dialect-driven: - * CASE — a bare mixed-case short is folded by the server while the - reference is quoted (the DEV-1756 defect), and quoting a lowercase - one breaks the mirror image on upper-folding backends. - * RESERVED / UNSAFE — ``index``/``int``/``rows`` are reserved in - MySQL but not in ``SLAYER_RESERVED_KEYWORDS``; the reference side - quotes them and a bare ``AS index`` is a syntax error. Checking - SLayer's set alone is not enough. - """ + """``AS `` spelling must match a downstream reference on every dialect (case + reserved).""" gen = SQLGenerator(dialect=dialect) ident = exp.Identifier(this=short, quoted=False) SQLGenerator._maybe_quote_ident(ident) @@ -898,9 +760,7 @@ def test_short_spelling_matches_the_reference_on_every_dialect( ) async def test_lowercase_short_stays_bare(self, chain) -> None: - """The Snowflake/Oracle mirror image: an all-lowercase short must NOT - be quoted, or the case-sensitive definition stops matching the bare - reference that those backends fold to upper.""" + """Mirror image: an all-lowercase short must stay bare on upper-folding backends.""" engine, _ = chain vm = await engine._query_as_model(inner_query=_repro_query()) lower = [ @@ -915,8 +775,6 @@ async def test_lowercase_short_stays_bare(self, chain) -> None: ) async def test_inner_and_wrapper_agree_on_the_fitted_alias(self, chain) -> None: - """The inner SQL is fitted by ``generate()``; the wrapper references - those aliases. They must not drift.""" engine, _ = chain vm = await engine._query_as_model(inner_query=_repro_query()) fitted = get_dialect("postgres").fit_alias(LONG_EMAIL) @@ -932,9 +790,7 @@ async def test_inner_and_wrapper_agree_on_the_fitted_alias(self, chain) -> None: assert LONG_EMAIL not in vm.sql async def test_case_colliding_shorts_raise(self, tmp_path) -> None: - """``email`` and ``Email`` on the deepest model produce shorts that - differ only by case. They are emitted into a namespace Postgres - case-folds, so this must be caught, not silently collapsed.""" + """``email`` and ``Email`` yield shorts differing only by case; must be caught.""" engine, _ = await _build_engine(tmp_path, case_colliding_columns=True) query = SlayerQuery( source_model="SandboxInvoiceV2", @@ -948,21 +804,7 @@ async def test_case_colliding_shorts_raise(self, tmp_path) -> None: await engine._query_as_model(inner_query=query) async def test_two_dimensions_landing_on_one_short_raise(self, tmp_path) -> None: - """Two DIMENSIONS whose shorts are exactly equal must be caught. - - `SandboxSubscription.SandboxCustomer.SandboxConsumer.name` flattens to - `SandboxSubscription__SandboxCustomer__SandboxConsumer__name`, which a - root column may also be named literally (column names permit `__`). - Both become `Column.name` on the virtual model. - - Enrichment's own guard does NOT cover this: `_occupied_shorts` is - populated from dimensions but only checked against MEASURES, so - dimension-vs-dimension slips through. It also flattens without length - fitting, so two shorts that differ before fitting and collide after it - would pass there too. Hence the check at the emission boundary — and it - has to key on the owning alias, since comparing the two shorts to each - other finds them equal and waves the pair through. - """ + """Two dimensions whose shorts are exactly equal must be caught at the emission boundary.""" flat = "SandboxSubscription__SandboxCustomer__SandboxConsumer__name" engine, _ = await _build_engine(tmp_path, decoy_root_column=flat) query = SlayerQuery( @@ -976,14 +818,7 @@ async def test_two_dimensions_landing_on_one_short_raise(self, tmp_path) -> None assert flat in str(exc.value) async def test_caller_supplied_measure_names_are_fitted(self, chain) -> None: - """A measure/transform/expression ``name`` is caller-supplied and - bypasses ``_alias_to_short``, so it needs its own fitting. - - These land in ``column_map`` verbatim, which means they reach the SQL as - ``AS ""`` AND become ``Column.name``. Two over-limit names sharing - a 63-byte prefix are truncated onto one column by Postgres, and the - ``short_owner`` check cannot see it — they differ as Python strings. - """ + """A caller-supplied measure ``name`` bypasses ``_alias_to_short`` and needs its own fitting.""" engine, _ = chain long_a = "z" * 63 + "b" # 64 bytes long_b = "z" * 63 + "c" # 64 bytes, identical first 63 @@ -1000,7 +835,7 @@ async def test_caller_supplied_measure_names_are_fitted(self, chain) -> None: names = [c.name for c in vm.columns] for name in names: assert _nbytes(name) <= 63, f"{name!r} is {_nbytes(name)} bytes" - # The real defect: distinct AFTER the server's 63-byte truncation. + # The real defect: distinct after the server's 63-byte truncation. truncated = [n.encode()[:63] for n in names] assert len(set(truncated)) == len(truncated), ( f"two shorts collapse onto one 63-byte name: {names}" @@ -1025,9 +860,7 @@ async def test_nested_dag_two_levels_agree(self, chain) -> None: _assert_order_by_refs_resolve(resp.sql) -# =========================================================================== -# 7. Engine-level contract — consumers never see the shortened form -# =========================================================================== +# Engine-level contract: consumers never see the shortened form class TestEngineContract: @@ -1052,11 +885,7 @@ async def test_attribute_keys_are_result_keys(self, chain) -> None: async def test_get_column_types_decodes_fitted_aliases( self, chain, monkeypatch: pytest.MonkeyPatch, ) -> None: - """``get_column_types`` probes with generated SQL, so its metadata keys - are the EMITTED (fitted) aliases while the lookup below uses the - canonical ``EnrichedMeasure.alias``. Without a decode pass an - over-limit measure silently drops out of the type map. - """ + """``get_column_types`` probe keys are fitted aliases and must decode back to canonical.""" engine, _ = chain over_limit = "totalAmount_" + "x" * 60 # 72-char measure name @@ -1064,7 +893,7 @@ async def test_get_column_types_decodes_fitted_aliases( class _FakeClient: async def get_column_types(self, sql: str) -> dict[str, str]: - # Echo back what a server would: keys exactly as emitted. + # Echo back server keys exactly as emitted. for name, _ in _projection_aliases( next(iter(sqlglot.parse_one(sql, dialect="postgres").find_all(exp.Select))) ): @@ -1099,15 +928,11 @@ async def aclose(self) -> None: # pragma: no cover — not reached ) -# =========================================================================== -# 8. Sweep — every generator shape, not just the reported one -# =========================================================================== +# Sweep: every generator shape, not just the reported one class TestSweep: - """Aliases can be synthesized in the generator rather than stored on the - enriched buckets. Assert PAIRING (no canonical over-limit alias string - survives) across every SQL shape the generator can build.""" + """Assert pairing (no canonical over-limit alias survives) across every SQL shape.""" @pytest.fixture def queries(self) -> list[SlayerQuery]: diff --git a/tests/test_query_backed_models.py b/tests/test_query_backed_models.py index 15e64386..33f0e593 100644 --- a/tests/test_query_backed_models.py +++ b/tests/test_query_backed_models.py @@ -1418,11 +1418,8 @@ async def test_inner_stage_aggregated_measure_honors_user_name(self) -> None: f"expected 'rev_sum' in cached columns, got: {col_names}" ) sql = loaded.backing_query_sql or "" - # Inner-stage wrap renames `"orders.rev" AS rev`; loose match on - # the alias keyword + name (newline-tolerant). DEV-1756: an - # all-lowercase short stays BARE so it folds the same way the - # downstream bare reference does; only mixed-case / reserved shorts - # are quoted. + # Inner-stage wrap renames `"orders.rev" AS rev`; an all-lowercase + # short stays bare so it folds like the bare downstream reference. import re assert re.search(r"\bAS\s+rev\b", sql), ( f"expected inner-stage 'AS rev' rename in SQL:\n{sql}" From f3b70bef6fd9efaba975f902f2b0a01cc7852398 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Tue, 11 Aug 2026 18:00:27 +0200 Subject: [PATCH 10/10] docs(DEV-1756): cut the identifier-length section to a short note Reduce the database-support.md section from ~60 lines to a 7-line note: some DBs cap identifier length (Postgres 63 bytes, silent), SLayer trims over-limit aliases, output column names are unaffected, only the introspectable SQL shows the trimmed form. Co-Authored-By: Claude Opus 4.8 --- docs/database-support.md | 65 +++++----------------------------------- 1 file changed, 7 insertions(+), 58 deletions(-) diff --git a/docs/database-support.md b/docs/database-support.md index a33be99c..f97b711c 100644 --- a/docs/database-support.md +++ b/docs/database-support.md @@ -37,64 +37,13 @@ Oracle. ## Identifier length limits -SLayer's result-column aliases are model-qualified join paths -(`orders.customers.regions.name`), so on a deep join chain they can grow past -what the database allows for an identifier. Postgres is the tightest of the -Tier-1 set at **63 bytes**, and — unlike every other engine here — it -**truncates silently**, emitting only a `NOTICE`. Two aliases sharing a 63-byte -prefix therefore become the same output column: the query either fails with -`AmbiguousColumnError` or, with no sibling to collide with, quietly returns the -column under a name nothing looks up. - -Each dialect declares a budget as `SqlDialect.max_identifier_bytes`. Anything -longer is shortened at emission to `__` — a deterministic -form that keeps both the root model and the column name readable, with an -8-hex-character SHA-256 of the full original in between: - -``` -orders.customers.regions.districts.neighbourhoods.neighbourhood_name (68 bytes) - -> orders.customers.regions.di_e6932600_urhoods.neighbourhood_name (63 bytes) -``` - -Two properties matter for consumers: - -- **Result keys never change.** The shortening is reversed on the way back, so - `response.data` and `response.columns` always carry the canonical dotted - alias regardless of which backend ran the query. Only `response.sql` (and - `dry_run` / `EXPLAIN` output) shows the shortened form, because that is the - SQL that actually executed. -- **Nothing is shortened unless it must be.** An alias that already fits is - emitted byte-for-byte as before, so SQL stays readable on every engine and - under-limit output is unchanged. - -| Dialect | Budget (bytes) | Over-limit behaviour | -|---|---|---| -| Postgres | 63 | **silent truncation** | -| MySQL | 64 | error | -| Redshift | 127 | error | -| Oracle, SQL Server | 128 | error | -| Snowflake | 255 | error | -| DuckDB | 256 | error | -| BigQuery | 300 | error | -| SQLite, ClickHouse, Trino/Presto, Databricks/Spark | unbounded | — | - -The budget is one conservative number per dialect rather than a full model of -each engine's per-identifier-class rules. It is counted in **bytes**, which is -conservative for engines that count characters (MySQL, SQL Server). MySQL's -64-character *identifier* limit is used rather than its more generous 256-char -*column alias* limit, and Oracle assumes 12.2+ (128 bytes; the pre-12.2 limit -of 30 is not modelled). A dialect added without setting the field inherits the -tightest value, since over-shortening is safe and under-shortening is not. - -Two SLayer-generated identifiers colliding after shortening raises -`IdentifierCollisionError` rather than emitting ambiguous SQL. With a 32-bit -digest over the full original this is astronomically unlikely; the check exists -because a duplicate output name on a silently-truncating backend is exactly the -failure this machinery prevents. - -Join-path *table* aliases (`customers__regions`) are not yet length-bounded — -they are internal and shorter, but a sufficiently deep chain of long model -names can still collide. Tracked separately. +Some databases cap identifier length (Postgres is the tightest at 63 bytes, and +truncates over-limit names *silently*). SLayer's join-path aliases like +`orders.customers.regions.name` can exceed that on a deep chain, so any +over-limit alias is trimmed at emission to `__`. This does +**not** affect output column names — `response.data` / `response.columns` keep +the canonical dotted alias — but `response.sql` (and `dry_run` / `EXPLAIN`) +shows the trimmed form, so account for it if you introspect the SQL. ## Aggregation support