diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 26af9d5c..a08c9ea6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -214,3 +214,53 @@ jobs: - name: Wipe Snowflake credentials if: always() run: rm -f ~/.snowflake/connections.toml + + redshift-integration: + # Skip on forked-PR runs where the Redshift secrets aren't available — the + # job would fail otherwise and we'd rather not nag external contributors. + if: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }} + runs-on: ubuntu-latest + needs: lint-and-test + steps: + - uses: actions/checkout@v4 + + - name: Skip if Redshift secrets are unset + id: gate + env: + REDSHIFT_HOST: ${{ secrets.REDSHIFT_HOST }} + run: | + if [[ -z "$REDSHIFT_HOST" ]]; then + echo "Redshift secrets unset — skipping the integration suite." + echo "skip=true" >> "$GITHUB_OUTPUT" + else + echo "skip=false" >> "$GITHUB_OUTPUT" + fi + + - name: Set up Python + if: steps.gate.outputs.skip != 'true' + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install Poetry + if: steps.gate.outputs.skip != 'true' + run: pip install poetry + + - name: Install dependencies (with redshift extra) + if: steps.gate.outputs.skip != 'true' + run: poetry install -E redshift + + - name: Run Redshift integration tests + if: steps.gate.outputs.skip != 'true' + timeout-minutes: 15 + # Points at a persistent dev cluster/serverless endpoint (unlike the + # Docker-backed suites, there's no ephemeral-container option for + # Redshift) — tests create and drop their own uniquely-named schema, + # see tests/integration/test_integration_redshift.py. + env: + REDSHIFT_HOST: ${{ secrets.REDSHIFT_HOST }} + REDSHIFT_PORT: ${{ secrets.REDSHIFT_PORT }} + REDSHIFT_DATABASE: ${{ secrets.REDSHIFT_DATABASE }} + REDSHIFT_USER: ${{ secrets.REDSHIFT_USER }} + REDSHIFT_PASSWORD: ${{ secrets.REDSHIFT_PASSWORD }} + run: poetry run pytest tests/integration/test_integration_redshift.py -v -m integration --timeout=300 diff --git a/DECISIONS.md b/DECISIONS.md index 496c8b6f..17aaaf0b 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -69,3 +69,4 @@ implementation detail. Include issue refs when known. - 2026-08-03 — Optional blocks + Cube JS/FILTER_PARAMS import (DEV-1730 / #270): a Mode-A-only `{? ... ?}` block renders its content parenthesised when every inner `{var}` is supplied, else collapses to the neutral `(1=1)` — the SLayer form of a Cube `FILTER_PARAMS` optional pushdown. Blocks live in the same `substitute_variables` (escape="sql") scanner as `{var}`/`{{`/`}}`, must contain ≥1 var, do not nest, and are rejected in Mode-B. A block-bearing model runs substitution even on a zero-variable call so its blocks collapse (the `_substitute_model_sql_surfaces` fast-path now checks for `{?` too); a block-free, required-only model with zero variables is still left untouched (the documented DEV-1625 raw-brace-literal boundary). `extract_model_variables(model)` derives required (bare, no default) vs optional (in-block or defaulted) from the four Mode-A surfaces — structural, nothing persisted, surfaced additively in the inspect skeleton `Variables:` line. The Cube importer gains a **JavaScript front-end** (esprima ESTree parser, a new core dep) that parses `cube()`/`view()` into the same `CubeCube`/`CubeView` shapes as YAML (dynamic values → report + skip member). FILTER_PARAMS refs are carried JS→converter as structured `CubeFilterParamRef` on the transient `CubeCube` (sentinels in the surface text; no arrow-body re-parse, sidestepping the `{var}`-vs-`{FILTER_PARAMS…}` brace clash); the converter resolves sentinels AFTER `translate_cube_refs` so the introduced `{var}` are never eaten. Requiredness (bare vs block) is decided in the converter alone via `honor_required_meta` (default on; CLI `--ignore-required-meta`) AND the member's `meta.required`; with the flag off a scalar-position arrow collapses to Cube's own `(1=1)::TIMESTAMP` booby-trap, faithfully. Cross-cube refs, unknown members, and generated-name collisions (`d`→`d_from` clashing member `d_from`) drop the cube (`filter_params_unsupported`); each logical variable is reported once (`filter_params_variable`) and stashed in `meta.cube_variables`. `render_probe_text` (blocks→`(1=1)`, bare vars→`0`) is the single import-time validation renderer, matching runtime collapse. - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. +- 2026-08-05 — Redshift live connector (`redshift-live-connector` branch): manual pass against a personal Redshift Serverless endpoint does not promote Redshift from Tier 2 to Tier 1 — the tier's meaning is CI-enforced non-regression, and no CI job runs it yet. Deliberately not wiring `REDSHIFT_*` secrets into `redshift-integration` CI until a project-owned (non-personal) endpoint exists: the job's fork-PR skip does not gate same-repo pushes/PRs, so pointing it at a personal AWS resource would bill every contributor's CI run to that account and expose the credential to anything running in that job. diff --git a/docs/configuration/datasources.md b/docs/configuration/datasources.md index 33f424da..77cf5565 100644 --- a/docs/configuration/datasources.md +++ b/docs/configuration/datasources.md @@ -64,12 +64,12 @@ These databases are verified by integration tests and runnable Docker examples. #### Additional support -SQL generation is covered by unit tests, but not verified against live instances. Install the appropriate SQLAlchemy driver manually. +SQL generation is covered by unit tests, but not verified against live instances. | Type | SQLAlchemy Driver | Install | |------|-------------------|---------| | `bigquery` | `sqlalchemy-bigquery` | `pip install sqlalchemy-bigquery` | -| `redshift` | `sqlalchemy-redshift` + `redshift_connector` | `pip install sqlalchemy-redshift redshift-connector` | +| `redshift` | `sqlalchemy-redshift` + `redshift-connector` | `pip install 'motley-slayer[redshift]'`. Connection-layer code is unit-tested and has passed manually against a live Redshift Serverless endpoint (outside CI) — see [Redshift](#redshift) below. | | `trino` / `presto` / `athena` | `trino` or `PyAthena` | `pip install trino` or `pip install PyAthena` | | `databricks` / `spark` | `databricks-sql-connector` | `pip install databricks-sql-connector` | | `oracle` | `oracledb` | `pip install oracledb` | @@ -148,6 +148,57 @@ Statement-level timeout is enforced via !!! tip If your database isn't listed but is supported by sqlglot, it may already work — SLayer falls back to Postgres-style SQL by default. Try it and [open an issue](https://github.com/MotleyAI/slayer/issues) if you hit a problem. +### Redshift + +```bash +pip install 'motley-slayer[redshift]' +``` + +The extra pulls in `sqlalchemy-redshift` + `redshift-connector`. Structured +`DatasourceConfig` fields (`host`, `port`, `database`, `username`, `password`) +build a `redshift+redshift_connector://` URL by default: + +```yaml +# datasources/rs.yaml +name: rs +type: redshift +host: mycluster.abc123.us-east-1.redshift.amazonaws.com +port: 5439 +database: dev +username: YOUR_USER +password: YOUR_PASSWORD +``` + +`redshift-connector` is AWS's actively-maintained driver — it supports IAM +auth, Redshift Serverless, and browser-based SSO in addition to plain +username/password. To use the classic psycopg2-based dialect instead (e.g. +for parity with an existing psycopg2 connection pool), set `connection_string` +directly rather than the structured fields. This requires `psycopg2-binary`, +which the `redshift` extra does not install — pull it in via the `postgres` +extra or install it separately: + +```bash +pip install 'motley-slayer[redshift,postgres]' +``` + +```yaml +name: rs +connection_string: "redshift+psycopg2://user:pass@host:5439/dev" +``` + +!!! warning "No native FK-based auto-ingestion join discovery (unverified)" + Redshift allows declaring (unenforced) `FOREIGN KEY` constraints, but + whether `sqlalchemy-redshift`'s Inspector surfaces them for auto-ingestion + to discover joins from — the way it does for Postgres / MySQL / SQLite / + Snowflake — has not been verified against a live cluster. Until confirmed, + treat Redshift like BigQuery/ClickHouse: define `joins:` manually in your + model YAML rather than relying on auto-discovery. + +!!! note "`APPROXIMATE COUNT(DISTINCT x)`" + Redshift's `count_distinct_approx` aggregation uses the keyword-prefix + form `APPROXIMATE COUNT(DISTINCT x)` rather than a native aggregate + function name (see [Aggregation support](../database-support.md)). + ## Field Reference | Field | Type | Required | Description | diff --git a/docs/database-support.md b/docs/database-support.md index fa652472..6b9238df 100644 --- a/docs/database-support.md +++ b/docs/database-support.md @@ -32,8 +32,23 @@ testcontainers suites. Unit tests for SQL generation; no live-instance verification. -Redshift, Trino/Presto (Athena uses the Presto dialect), Databricks/Spark, -Oracle. +Trino/Presto (Athena uses the Presto dialect), Databricks/Spark, Oracle. + +**Redshift** is further along than the rest of this tier: `RedshiftDialect` +(SQL generation) has been code-covered here for a while, and the connection +layer — `driver_map["redshift"] = "redshift+redshift_connector"`, the +`motley-slayer[redshift]` extra, and a live integration suite +(`tests/integration/test_integration_redshift.py`, gated on `REDSHIFT_HOST` / +`REDSHIFT_DATABASE` / `REDSHIFT_USER` / `REDSHIFT_PASSWORD`) — now exist too. +The suite has passed manually against a live Redshift Serverless endpoint +(outside CI). It stays in Tier 2 rather than Tier 1 because the +`redshift-integration` CI job (`.github/workflows/ci.yml`) has not yet run it +against a live endpoint — CI secrets are deliberately not wired up yet, since +the only endpoint verified so far is a personal AWS resource, and pointing CI +at it would bill every contributor's CI run to that personal account. This +entry should move to Tier 1 once the CI job passes at least once against a +project-owned (non-personal) Redshift endpoint, the same skip-until-configured +way Snowflake/BigQuery started. ## Aggregation support diff --git a/examples/redshift/README.md b/examples/redshift/README.md new file mode 100644 index 00000000..2f3bf0a5 --- /dev/null +++ b/examples/redshift/README.md @@ -0,0 +1,84 @@ +# Redshift example + +Redshift is a Tier 2 dialect (see [database support](../../docs/database-support.md)): +`RedshiftDialect`'s SQL generation is unit-tested, and the connection layer added +alongside this example (`motley-slayer[redshift]`, `driver_map["redshift"]`) is +too. The connection layer has since passed manually against a live Redshift +Serverless endpoint (outside CI) — see the CI note under Known limitations. +Like Snowflake / BigQuery, there's no free local Docker image, so this example +needs a real cluster or Redshift Serverless workgroup. + +## 1. Install the extra + +```bash +pip install 'motley-slayer[redshift]' +``` + +The extra pulls in `sqlalchemy-redshift` + `redshift-connector`. + +## 2. Configure a connection + +```yaml +# slayer_data/datasources/rs.yaml +name: rs +type: redshift +host: mycluster.abc123.us-east-1.redshift.amazonaws.com +port: 5439 +database: dev +username: YOUR_USER +password: YOUR_PASSWORD +``` + +Or as a connection string, using `redshift-connector` (the default driver — +supports IAM auth and Redshift Serverless, not just password auth): + +```text +redshift+redshift_connector://YOUR_USER:YOUR_PASSWORD@mycluster.abc123.us-east-1.redshift.amazonaws.com:5439/dev +``` + +## 3. Seed the demo schema + +```bash +python ../seed.py "redshift+redshift_connector://YOUR_USER:YOUR_PASSWORD@mycluster.abc123.us-east-1.redshift.amazonaws.com:5439/dev" +``` + +This drops + recreates the four canonical tables (`regions`, `customers`, +`products`, `orders`) and inserts the standard fixture dataset. + +## 4. Register the datasource and ingest + +```bash +slayer datasources create "redshift+redshift_connector://YOUR_USER:YOUR_PASSWORD@.../dev" --name rs --ingest +``` + +**Redshift allows declaring `FOREIGN KEY` constraints, but they're not +enforced, and whether they're reflected for auto-ingestion's join discovery +is unverified against a live cluster.** Treat this like the ClickHouse / +BigQuery examples rather than the Postgres / MySQL / Snowflake ones: assume +joins are **not** auto-generated, and add them manually to the model YAML +files under `slayer_data/models/` if you need cross-model rollups. + +## 5. Verify + +```bash +python verify.py +``` + +`verify.py` runs the same battery used by the other examples: schema/type +checks, aggregation matrix (`median`, `percentile`, `stddev_samp/pop`, +`var_samp/pop`, `corr`, `covar_samp/pop`), and `count_distinct_approx` — which +on Redshift compiles to the keyword-prefix form `APPROXIMATE COUNT(DISTINCT x)` +rather than a native aggregate function name. + +## Known limitations + +- **No auto-discovered joins** (see step 4) — unverified either way; add + `joins:` manually until confirmed. +- **Not yet CI-verified.** `tests/integration/test_integration_redshift.py` + has passed manually against a live Redshift Serverless endpoint, but the + `redshift-integration` CI job still skips — `REDSHIFT_HOST` / + `REDSHIFT_DATABASE` / `REDSHIFT_USER` / `REDSHIFT_PASSWORD` repo secrets are + deliberately not configured yet, since the only endpoint verified so far is + a personal one and wiring it into CI would bill every contributor's run to + that account. Redshift moves to Tier 1 once the job passes against a + project-owned endpoint. diff --git a/examples/redshift/verify.py b/examples/redshift/verify.py new file mode 100644 index 00000000..287b69b6 --- /dev/null +++ b/examples/redshift/verify.py @@ -0,0 +1,80 @@ +"""Verification script for the Redshift example. + +Run after seeding + ingesting: + python examples/seed.py "redshift+redshift_connector://user:pass@host:5439/dev" + slayer datasources create "redshift+redshift_connector://user:pass@host:5439/dev" --name rs --ingest + python examples/redshift/verify.py + +Redshift's FOREIGN KEY constraints are declarative-only (not enforced), and +whether sqlalchemy-redshift's Inspector surfaces them for auto-ingestion join +discovery is unverified against a live cluster — treated here like +ClickHouse/BigQuery (no rollup joins expected), not like Postgres/MySQL/ +Snowflake. +""" + +import os +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) +from verify_common import ( + check, + check_column_types, + check_corr_covar, + check_median_percentile, + check_rollup, + check_stddev_var, + run_common_checks, + summary, +) + +if __name__ == "__main__": + models = run_common_checks() + check("4 models (no rollup)", len(models) == 4) + check_rollup(expect_rollup=False) + # BIGINT -> INT; VARCHAR -> TEXT; DECIMAL(10,2) -> DOUBLE; TIMESTAMP -> TIMESTAMP. + check_column_types( + model_name="orders", + expected_types={ + "id": "INT", + "customer_id": "INT", + "product_id": "INT", + "quantity": "INT", + "status": "TEXT", + "created_at": "TIMESTAMP", + }, + ) + check_column_types( + model_name="customers", + expected_types={ + "id": "INT", + "name": "TEXT", + "email": "TEXT", + "region_id": "INT", + }, + ) + check_column_types( + model_name="products", + expected_types={ + "id": "INT", + "name": "TEXT", + "category": "TEXT", + "price": "DOUBLE", + }, + ) + check_column_types( + model_name="regions", + expected_types={ + "id": "INT", + "name": "TEXT", + }, + ) + # RedshiftDialect (slayer/sql/dialects/_tier2.py) only overrides + # build_approx_count_distinct — median/percentile/stddev/var/corr/covar + # all fall through to the shared Postgres-shaped SqlDialect base, same as + # every other Tier-2 dialect. Whether that's Redshift's *native* syntax + # or a formula fallback is unconfirmed without a live cluster; this just + # checks the queries execute and return sane values either way. + check_median_percentile() + check_stddev_var() + check_corr_covar() + summary() diff --git a/poetry.lock b/poetry.lock index 537300cc..d0b95228 100644 --- a/poetry.lock +++ b/poetry.lock @@ -358,7 +358,7 @@ description = "Fast ASN.1 parser and serializer with definitions for private key optional = true python-versions = "*" groups = ["main"] -markers = "extra == \"snowflake\" or extra == \"all\"" +markers = "extra == \"snowflake\" or extra == \"all\" or extra == \"redshift\"" files = [ {file = "asn1crypto-1.5.1-py2.py3-none-any.whl", hash = "sha256:db4e40728b728508912cbb3d44f19ce188f218e9eba635821bb4b68564f8fd67"}, {file = "asn1crypto-1.5.1.tar.gz", hash = "sha256:13ae38502be632115abf8a24cbe5f4da52e3b5231990aff31123c805306ccb9c"}, @@ -521,11 +521,12 @@ version = "4.14.3" description = "Screen-scraping library" optional = false python-versions = ">=3.7.0" -groups = ["dev"] +groups = ["main", "dev"] files = [ {file = "beautifulsoup4-4.14.3-py3-none-any.whl", hash = "sha256:0918bfe44902e6ad8d57732ba310582e98da931428d231a5ecb9e7c703a735bb"}, {file = "beautifulsoup4-4.14.3.tar.gz", hash = "sha256:6292b1c5186d356bba669ef9f7f051757099565ad9ada5dd630bd9de5fa7fb86"}, ] +markers = {main = "extra == \"redshift\" or extra == \"all\""} [package.dependencies] soupsieve = ">=1.6.1" @@ -564,7 +565,7 @@ description = "The AWS SDK for Python" optional = true python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"snowflake\" or extra == \"all\"" +markers = "extra == \"snowflake\" or extra == \"all\" or extra == \"redshift\"" files = [ {file = "boto3-1.43.27-py3-none-any.whl", hash = "sha256:b3eea072c2fdbbdd8c6161f912f603be10c8ec477625926dea8b91a0842a3482"}, {file = "boto3-1.43.27.tar.gz", hash = "sha256:dc0d1b47f391983d8b3047e49402d31f9aaa4d7b398d3b4ea986fe680cbea43a"}, @@ -585,7 +586,7 @@ description = "Low-level, data-driven core of boto 3." optional = true python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"snowflake\" or extra == \"all\"" +markers = "extra == \"snowflake\" or extra == \"all\" or extra == \"redshift\"" files = [ {file = "botocore-1.43.27-py3-none-any.whl", hash = "sha256:4976544e652d5a1d8eca135da019f8e1c2d749efa2f9a31a8fb8c76f1895a40b"}, {file = "botocore-1.43.27.tar.gz", hash = "sha256:2093c316c24214e50e18640b1869513b759bb8cc48b95b004a8306cb9f0d6703"}, @@ -859,7 +860,7 @@ files = [ {file = "charset_normalizer-3.4.7-py3-none-any.whl", hash = "sha256:3dce51d0f5e7951f8bb4900c257dad282f49190fdbebecd4ba99bcc41fef404d"}, {file = "charset_normalizer-3.4.7.tar.gz", hash = "sha256:ae89db9e5f98a11a4bf50407d4363e7b09b31e55bc117b4f7d80aab97ba009e5"}, ] -markers = {main = "(extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"clickhouse\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\") or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\""} +markers = {main = "(extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"redshift\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"clickhouse\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"redshift\") or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"redshift\""} [[package]] name = "ciso8601" @@ -2980,7 +2981,7 @@ description = "JSON Matching Expressions" optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"snowflake\" or extra == \"all\"" +markers = "extra == \"snowflake\" or extra == \"all\" or extra == \"redshift\"" files = [ {file = "jmespath-1.1.0-py3-none-any.whl", hash = "sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64"}, {file = "jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d"}, @@ -3522,6 +3523,157 @@ proxy = ["PyJWT (>=2.12.0,<3.0.0) ; python_version >= \"3.9\"", "apscheduler (>= semantic-router = ["semantic-router (>=0.1.12) ; python_version >= \"3.9\" and python_version < \"3.14\""] utils = ["numpydoc"] +[[package]] +name = "lxml" +version = "6.1.1" +description = "Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API." +optional = true +python-versions = ">=3.8" +groups = ["main"] +markers = "extra == \"redshift\" or extra == \"all\"" +files = [ + {file = "lxml-6.1.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:09dd5b7075dc2f7709654a46543ba1ea3c2e217b2ed8fbd413a8a945a0f40f60"}, + {file = "lxml-6.1.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:f6ac4ef4d82dff54670227a69c67782ae0b811b5cf6b17954f1e8f7502fc0d1d"}, + {file = "lxml-6.1.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:556e94a63c9b04716f8e4de2abb65775061f846e89331b6c5be79183a24f98ea"}, + {file = "lxml-6.1.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5c6bf403fbb3b3e348a561a5f4f0b9961835657981c802a1df03653eef8a9074"}, + {file = "lxml-6.1.1-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1dde6131244bba38a17c745836ba190bc753fd73c9291666287fd0a3fa3dcf30"}, + {file = "lxml-6.1.1-cp310-cp310-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:98fc784c2c1440667aeedf8465bdfe10208acf0ead656a2c68627299f546b315"}, + {file = "lxml-6.1.1-cp310-cp310-manylinux_2_28_i686.whl", hash = "sha256:add8cf6ddf9a65116119a28ece0f7886e30af27ba724a7594305f1d1b58a92a1"}, + {file = "lxml-6.1.1-cp310-cp310-manylinux_2_31_armv7l.whl", hash = "sha256:cf9d57306d848218f3601fee7601fab1a327c942d56e2e97610583cb4dd74206"}, + {file = "lxml-6.1.1-cp310-cp310-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:88136950da4d13c318bde414ce10219931937851327f44328f2df4d2c4614067"}, + {file = "lxml-6.1.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:cecdd5dfdc87b1fd87dbf81d4b037a544f47f4c744200a67013771682d67686a"}, + {file = "lxml-6.1.1-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:cd312b9692e831d2ffcad61eab31d91d4b4655a962e61de8fb410472cbcd37aa"}, + {file = "lxml-6.1.1-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:5b7328b46d49fc9477d91ae8f6d55340347d827b7734ba3ea33faae0efef1383"}, + {file = "lxml-6.1.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:37a58976370f36d9329d118ad0b953c5aeb9119ac9c6a4e258942a225d0573a1"}, + {file = "lxml-6.1.1-cp310-cp310-win32.whl", hash = "sha256:cea3f4c1af79af13cdb2da0c028111d8f8522d4f22a000c82385535f24e5cf3a"}, + {file = "lxml-6.1.1-cp310-cp310-win_amd64.whl", hash = "sha256:3abf332af33a74288675d936fe861fd4344da0dd6622193fbc4f2bfbb35536b5"}, + {file = "lxml-6.1.1-cp310-cp310-win_arm64.whl", hash = "sha256:8dadbe5b217ff35b6a8d16610dd710219b59b76d13f0e3f0d9f36786206e4485"}, + {file = "lxml-6.1.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:53b7d2b7a10b1c35c0a5e21e9224accf60c1bbfba523990732e521b2b73adef2"}, + {file = "lxml-6.1.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ff3f333630ab480244a1bff72043e511a91eb22e7595dead8653ee5612dd8f3d"}, + {file = "lxml-6.1.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a4bbea04c97f6d78a48e3fbc1cb9116d2780b1b39e03a23f6eb9b603fd61f510"}, + {file = "lxml-6.1.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:db1d75f6617a49c1c01bc7023713e0ff59ab32c9579ae62a7674c0e34f3b0b0a"}, + {file = "lxml-6.1.1-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a12689be69a28ddaa0ab99a5a1137da2afd5f8f16df7b5680b66f616d3eda1d"}, + {file = "lxml-6.1.1-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:18b73c339ae29b90fd2d06e58ebd555a751bde9cd6bbd36cc0281b9a2c94e9d8"}, + {file = "lxml-6.1.1-cp311-cp311-manylinux_2_28_i686.whl", hash = "sha256:752d3bbfe874715ccd0aec7f88d7fc623c0f1fd7aa7b3238a084e017bad2a009"}, + {file = "lxml-6.1.1-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:6b1761fbf9ec984e2e9d9c589ef5f5fd684b7c19f92aadd567a26c5224958db6"}, + {file = "lxml-6.1.1-cp311-cp311-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d680fbcb768404c601ecb43519ecd8461f6954cb11c06a78962f666832ccfca8"}, + {file = "lxml-6.1.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:162af1091cd785f2f27e62d3547ae9bc58ec5c86dd314d67021fd02463708d83"}, + {file = "lxml-6.1.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:e9308ff8241c532df3f3e570f9a5aeed6c853f888512ba4b75638d7c11c95ef6"}, + {file = "lxml-6.1.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:5f6994074ebae6ffb04447268e37dc16edc304f9859cf91acb86e0af6c1b395c"}, + {file = "lxml-6.1.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:80c2dfadb855da477cf73373ad29a333535dedb9b12bad02c9814c8e2b43bf08"}, + {file = "lxml-6.1.1-cp311-cp311-win32.whl", hash = "sha256:30a89d3ac8faec007453fb541f3f46807eeec88edd5826f6e3fe001752a2c621"}, + {file = "lxml-6.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:abbefa31eee84842140f67acef1c828e28bba8bbf0c3bc6e5492a9af88152c28"}, + {file = "lxml-6.1.1-cp311-cp311-win_arm64.whl", hash = "sha256:dcb292aa7fe485ceff7af4f92e46c5af397daec5dff64871a528f0fc47a3cc5b"}, + {file = "lxml-6.1.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:104c09bda8d2a562824c0e319d0768ce26a779b7601e0931d33b09b53c392ef7"}, + {file = "lxml-6.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:25c6997a9a534e016695a0ba06b2f07945de682731ff01065b6d5a4474179da1"}, + {file = "lxml-6.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c921ba5c51e4e9f63b8b00267d06566e1f63407408a0496da2d1d0bfc819c7fc"}, + {file = "lxml-6.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:54a7f95e4de5fb94e2f9f4b9055c6ba33bf3d628fd77a1d647c5923caa2cdcdc"}, + {file = "lxml-6.1.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f2ec43df44b1f76249ee0a615334f9b5b060e1c8bd90e706dad2d14d02f383"}, + {file = "lxml-6.1.1-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:70ef8a7e102a1508f8121aae5b0867abd663f72c14f0a9c937e6554cb4587b7b"}, + {file = "lxml-6.1.1-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ebe6af670449830d6d9b752c256a983291c766a1365ba5d5460048f9e33a7818"}, + {file = "lxml-6.1.1-cp312-cp312-manylinux_2_28_i686.whl", hash = "sha256:27acc820660aaffa4f7c087f29120e12980f7779d56d8492d263170111284740"}, + {file = "lxml-6.1.1-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:1db753c9115ec7100d073b744d17e25e88a8f90f5c39b2f5dd878149af59671f"}, + {file = "lxml-6.1.1-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c4f469aebd783bb741c2ecb2a681008fd26bfe5c16a9a72ed5467f834e810df2"}, + {file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:766b010012d59470072c1816b5b6c69f1d243e5db36ea5968e94accf430a4635"}, + {file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b8d812c6011c08b8111a15e54dd990b8923692d80adf35488bee34026c35accf"}, + {file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:fe0306bd29505a9177aac19f1877174b0e7422c222a59f70b2cd41633448c3dc"}, + {file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:5ba186ad207446c65d3bb3d3e0412b032b1d9f595e59861e2354798c5703d955"}, + {file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:aa366a1e55b8ebfe8ca8ddc3cfe75c8ebade181aeb0f661d0cb05986b647f72a"}, + {file = "lxml-6.1.1-cp312-cp312-win32.whl", hash = "sha256:126c93f7f56f0eda92f6d8c619edc463a4f23d9252f1c9d0405a76f25fa9f11a"}, + {file = "lxml-6.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:26e6eda8d38c1fcab1090dd196ee87cbd13788e531937610e2589085de074e77"}, + {file = "lxml-6.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:6540377fbd53fe1b629172288c464fb18db11ce1fa7dc15891da10aa9dcc3e7f"}, + {file = "lxml-6.1.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:68a9198d0fc122d14bb76837de9aa80cf84caed990b5b237f532ed87d3706736"}, + {file = "lxml-6.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7d47866cb32fb503450b6edc9df355d10dc49836af2e89901bd6ac6b0896d9d9"}, + {file = "lxml-6.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb7c9811bfaa8b1ed5ed319f5d370dfbcaa59d52ea64be2a5a85e18195930354"}, + {file = "lxml-6.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:762ff394d5bd56da0cf034a23dcce4e13923f15321a2adfa2ac00201dc6d3fca"}, + {file = "lxml-6.1.1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a088f287f7d8275a33c07f2cac6c50b9319309a0200a39e7e75d80c707723099"}, + {file = "lxml-6.1.1-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e902da4b04e6b52e5893900d4b8ab46068f75f3561f01bf1080957f9fd932ed6"}, + {file = "lxml-6.1.1-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1d4962d4c66bf830a7e59ed6cfc17d148149898a3aefa8ec6e59763e6e3ed085"}, + {file = "lxml-6.1.1-cp313-cp313-manylinux_2_28_i686.whl", hash = "sha256:581d4c8ae690a6609e64862dd6b7c2489635c2d13907fc2b20f2bc200ff1d21e"}, + {file = "lxml-6.1.1-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:876e1ff5930ed8bf295ec5ef9a8155e9b6b1876bbf1deed8b3a8069311875a8f"}, + {file = "lxml-6.1.1-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9eb9b5a968f6e0f6d640092a567e14529ff8cea2e29d00da6f78a79fa49f013c"}, + {file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:aa49e06d94aba782c6a02eecb7e507969e7e7a41b267f1b359bb35585f295d5b"}, + {file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:70cdfd80589d59e43e18005dd7244e8895e93db8ab6a620b7e23df5445a4e3d2"}, + {file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:aad9aa39483ed8ec44d6d2e59e5b98a0d80676ef0d92f44bfc374836111f62f5"}, + {file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:d49514be2f28d895c38cf9d2b72d7b9a07d00314519f456c0b50b53cfcf4c785"}, + {file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:47402e62c52ff5988c1e8c6c63177f5708bccf48e366dea4e3dcf1e645e04947"}, + {file = "lxml-6.1.1-cp313-cp313-win32.whl", hash = "sha256:3483644525531e1d5762b0c44a8e18b6efba321b6dcf8a8952de10b037618bca"}, + {file = "lxml-6.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:a10bd2fd62e8ce916ececb342f348f190724a098c1faa056fdfb2a22ad5e8660"}, + {file = "lxml-6.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:424aa57aca0897eb922aef34395bd1289b3b6f04e6bae20ea123c0c7e333cffc"}, + {file = "lxml-6.1.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:19b7ab10b210b0b3ad7985d9ac4eb66ab09a90b20fe6e2f7ba55d01a234345d0"}, + {file = "lxml-6.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c08e5c694306507275f2290073350c4f32e383db15213b2c69e7ff39c1193840"}, + {file = "lxml-6.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:74a9717fd0d82effef5c2854f0d917231d5324b5a3eb7275c43ac9fa32f97a14"}, + {file = "lxml-6.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:efe0374196335f93b53269acd811b944f2e6bdc88e8894f214bd636455484909"}, + {file = "lxml-6.1.1-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ac931cdc9442c1763b8a8f6cd62c0c938737eafc5be75eff88df55fc73bc0d00"}, + {file = "lxml-6.1.1-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:aee395f5d0927f947758b4ec119fd5fc8ec71f07a1c5c52077b30b04c0fa6955"}, + {file = "lxml-6.1.1-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9395002973c827b3ed67db77e6ec09f092919a587022174554096a269378fb13"}, + {file = "lxml-6.1.1-cp314-cp314-manylinux_2_28_i686.whl", hash = "sha256:73bc2086f141224ebddb7fc5c6a36ca58b31b94b561e1dfe8e073e3270fad1e7"}, + {file = "lxml-6.1.1-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:3779def59032b81e44a5f70096ef6bf2082f8d901937dca354474ba09782e245"}, + {file = "lxml-6.1.1-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:86c89b9d55ebf820ad7c90bc533410f0d098054f293351f10603c0c46ff598f5"}, + {file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19607c6bbff2a44cf3fe8250abccd20942d3462473e0a721d01d379ed017e462"}, + {file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:c6ed5141a5c7507cf3ee76bd363b0d6f801e3321adc35b5d825a23115faa5465"}, + {file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:62aeb7e85b5d60320b9d77eef2e773994e2c0ce10121b277e0a19804e1654a5a"}, + {file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:b1b963fd8f5caa68e99dfae060d54de1fe9cba899b8718b44a00cdca53c3e590"}, + {file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:63876be28efefa04a1df615b46770e82042cce445cfdce55160522f57b231ccb"}, + {file = "lxml-6.1.1-cp314-cp314-win32.whl", hash = "sha256:7f7a92e8583f06b1fd49d01158143b8461cfcd135dcb10ec807270a3051bd603"}, + {file = "lxml-6.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:b2d444f2e66624d68e9c6b211e28a76e22fff5fcabcfff4deac18b529b7d4137"}, + {file = "lxml-6.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:3fd9728a2735fda14f4e8235830c86b539e9661e849665bf926d3f867943b4bf"}, + {file = "lxml-6.1.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:787b2496d0dbe8cd180984e8d29e3a6f76e7ea34db781cb3bd55e4ba1ef8b4ee"}, + {file = "lxml-6.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:2c8daa471358dc2d6fcf02165e80ec68f77871a286df95bc5cc3816153b0fd2c"}, + {file = "lxml-6.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:acd7d70b64c0aae0c7922cca83d288a16f5f6da523637697872253415269baef"}, + {file = "lxml-6.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4f0dd2f01f9f8a89f565d000e03abcf0a13d692a346c8d22f628d49af098777a"}, + {file = "lxml-6.1.1-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b7e8a14c8634bf6f7a568634cb395305a6d964aeb5b7ee32248094bed3a7e2c"}, + {file = "lxml-6.1.1-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:86281fbdd6a8162756f8d603f37e3435bfa38043adb79c6dc6a2dfee065e7525"}, + {file = "lxml-6.1.1-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c5d7152ec39ca7c402d8fb9bad86140a15b9503bd0c54484e3f1bbe3dd37ceca"}, + {file = "lxml-6.1.1-cp314-cp314t-manylinux_2_28_i686.whl", hash = "sha256:88d8cb75b9d82858497a5393e3c63cfbf03035225e4b35a49ed7ccb151e4dc0e"}, + {file = "lxml-6.1.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f64ec5397ea6a41fc1b4af0380d79b44a755b5531dcaccd9940fb260dca93038"}, + {file = "lxml-6.1.1-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d34bbf07dbc7ca5970671b1512e928991fb5e9d95365636c9b2d8b4f53af405e"}, + {file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:17e0e18d4ad8adbd0399291bc44845b69d9dd68439a3cdebdf35ff902ec05072"}, + {file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:3ab541146f1f6968c462d6c2ac495148e8cdba2f8347700b2141b6ec5a75bf52"}, + {file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:2a0217714657e023ef4293500f65aa20fce6164c8fd6b08fa5bd4a859fb14b9b"}, + {file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:05a82eb6e1530a64f26225b55cbd178113bd0b5af1c2b625f25e5296742c26d2"}, + {file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9e36f163528fc50cbef305f02a5fd66d404edf7049cdaff211dbc2cba5a7013e"}, + {file = "lxml-6.1.1-cp314-cp314t-win32.whl", hash = "sha256:649dda677cf3bd6ac9ae14007ba0c824ded8ce5808b53fc7431d9140399118c1"}, + {file = "lxml-6.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:793033d6c5cdf33a573f910d9bea14ef8f5771820411d118da8e1182edb53d5e"}, + {file = "lxml-6.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:58bb955caba94e467d2a96da17660d2d704e0675894cba21ab8a775b8621fd1c"}, + {file = "lxml-6.1.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6689e828a94eee4f139408c337bb198e014724bb8a8c26d3cfac49d119ed69a6"}, + {file = "lxml-6.1.1-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bdebcc8a75d38c7598dfb2c9ed852d7a9eb4a10d6e2d0764b919b802bf32ac88"}, + {file = "lxml-6.1.1-cp38-cp38-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8be8ad51249698103d24b0571df35a10990fbe93dd043b6c024172189485f5e3"}, + {file = "lxml-6.1.1-cp38-cp38-manylinux_2_28_i686.whl", hash = "sha256:76447f65250ed2501ead1a1552f5ce8edff159a86f308348e6a9c4acb5e1f1b4"}, + {file = "lxml-6.1.1-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:ffecec8eb889b58ba9be5b95fb1cc78e22ea8eedea38e8736a1568fe1979250e"}, + {file = "lxml-6.1.1-cp38-cp38-win32.whl", hash = "sha256:c674693f055fa2495de12292cb45e9944199d8eaef5a2dec45175c7c61cb73e3"}, + {file = "lxml-6.1.1-cp38-cp38-win_amd64.whl", hash = "sha256:55b03549819867ea141c0202242c4816c82e52ec36e7e648db9d8da5a3dc3ed6"}, + {file = "lxml-6.1.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:c9f79d5325907f13e1be0b3e4dacc1049d1dffc4aeee3c995284bea5fe0fab7d"}, + {file = "lxml-6.1.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:83b6b30eb131da7a75b601f28c5d6971e6ed3e887919bf6b6a1ad3c2df289080"}, + {file = "lxml-6.1.1-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:441dd227fa0690eb9fc81edabc63cdcefc212bba99b906dcf6e32cc1a9d3e533"}, + {file = "lxml-6.1.1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e07c65f443c887bbcf31cc1771d932ecc192a5273943589b3c7572b749f1ffb2"}, + {file = "lxml-6.1.1-cp39-cp39-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5bec7d03d78d853597d6107854c2310ce3f761fd218fe9fe91d5101fcf6c2efe"}, + {file = "lxml-6.1.1-cp39-cp39-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9f76acfb5f68ba982635a53fd985a8044be98a35b43232c2a1ee235ffab3e1dd"}, + {file = "lxml-6.1.1-cp39-cp39-manylinux_2_28_i686.whl", hash = "sha256:8d43ca737b20e106e4aebc42b2f3ae19f00ba63d7eb731698ee083d72d15646f"}, + {file = "lxml-6.1.1-cp39-cp39-manylinux_2_31_armv7l.whl", hash = "sha256:32ab449a5486f6c758e849bb86710d0e45edc24a04e250c01555f8f5653958f8"}, + {file = "lxml-6.1.1-cp39-cp39-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:53c909b62a0532183542fed00c5a7218258c56292d409bc789886fe1cb04c438"}, + {file = "lxml-6.1.1-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:640f97d43d867bcb9c75b3af013b64850756b746cb6bce8ace83b70da3abba9d"}, + {file = "lxml-6.1.1-cp39-cp39-musllinux_1_2_armv7l.whl", hash = "sha256:469e3618338bd7ab5beb412d2439825479fcf0dab99e394ca563dbc4eaf6c834"}, + {file = "lxml-6.1.1-cp39-cp39-musllinux_1_2_riscv64.whl", hash = "sha256:aae97dfdb60715c164419ac2532a76d013c3918a665eb6cb7288098b5f349aaf"}, + {file = "lxml-6.1.1-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:c9a4b821dc7055bf9e05ff5719e18ec501f75c0f0bbfabd573b277559780833d"}, + {file = "lxml-6.1.1-cp39-cp39-win32.whl", hash = "sha256:639f6c857d91d9be29bd7502348d6736dab168b54b5158cd899abf11684dc186"}, + {file = "lxml-6.1.1-cp39-cp39-win_amd64.whl", hash = "sha256:34c2d737beabfe35baada43941ed519251e9a12e779031496bcd5d539fcfd730"}, + {file = "lxml-6.1.1-cp39-cp39-win_arm64.whl", hash = "sha256:07a4a68e286ee7a1ed7dfb8af83e615757c0ccfe9f18c6b4ea6771388d9ba8c9"}, + {file = "lxml-6.1.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:31033dc34636ea6b7d5cc11b1ddbda78a14de858ba9d3e1ed4b69a3085bc521e"}, + {file = "lxml-6.1.1-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3893c14c4b6ac5b2d54ba8cf03e99fe5104e592de491f19bd6b82756c09f8004"}, + {file = "lxml-6.1.1-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c07da4cebf6889f03ebac8d238f62318e29f495de0aa18a51ea14e61ae907e2e"}, + {file = "lxml-6.1.1-pp311-pypy311_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f6f0ce10945fab9c4c06ce14e22af9059d1a87493a9af4501a5b0b9187e21cf2"}, + {file = "lxml-6.1.1-pp311-pypy311_pp73-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f8844cd288697c6425c9beba919302241e3278871dc6519515e72b04e987abcf"}, + {file = "lxml-6.1.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:ed21202aec73cda4d55d1ce57b389aadb90ffb044e6cd1080b8347efe1b1ec84"}, + {file = "lxml-6.1.1.tar.gz", hash = "sha256:ba96ae44888e0185281e937633a743ea90d5a196c6000f82565ebb0580012d40"}, +] + +[package.extras] +cssselect = ["cssselect (>=0.7)"] +html-clean = ["lxml_html_clean"] +html5 = ["html5lib"] +htmlsoup = ["BeautifulSoup4"] + [[package]] name = "lz4" version = "4.4.5" @@ -5837,7 +5989,7 @@ files = [ {file = "pytz-2025.2-py2.py3-none-any.whl", hash = "sha256:5ddf76296dd8c44c26eb8f4b6f35488f3ccbf6fbbd7adee0b7262d43f0ec2f00"}, {file = "pytz-2025.2.tar.gz", hash = "sha256:360b9e3dbb49a209c21ad61809c7fb453643e048b38924c765813546746e81c3"}, ] -markers = {main = "extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"snowflake\""} +markers = {main = "extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"snowflake\" or extra == \"redshift\""} [[package]] name = "pywin32" @@ -6103,6 +6255,32 @@ numpy = "*" [package.extras] dev = ["pytest"] +[[package]] +name = "redshift-connector" +version = "2.1.15" +description = "Redshift interface library" +optional = true +python-versions = ">=3.8" +groups = ["main"] +markers = "extra == \"redshift\" or extra == \"all\"" +files = [ + {file = "redshift_connector-2.1.15-py3-none-any.whl", hash = "sha256:4e4b4b0e88a6f9dfe6c2ec9f81b328936ec1f32fc9549c6b960e1131364b3dd7"}, +] + +[package.dependencies] +beautifulsoup4 = ">=4.13.5,<5.0.0" +boto3 = ">=1.42.22,<2.0.0" +botocore = ">=1.12.201,<2.0.0" +lxml = ">=6.1.0,<7.0.0" +packaging = "*" +pytz = ">=2020.1" +requests = ">=2.23.0,<3.0.0" +scramp = ">=1.2.0,<1.5.0" +setuptools = "*" + +[package.extras] +full = ["numpy", "pandas"] + [[package]] name = "referencing" version = "0.37.0" @@ -6256,7 +6434,7 @@ files = [ {file = "requests-2.33.1-py3-none-any.whl", hash = "sha256:4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a"}, {file = "requests-2.33.1.tar.gz", hash = "sha256:18817f8c57c6263968bc123d237e3b8b08ac046f5456bd1e307ee8f4250d3517"}, ] -markers = {main = "(extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"clickhouse\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\") or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\""} +markers = {main = "(extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"redshift\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"clickhouse\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"redshift\") or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"redshift\""} [package.dependencies] certifi = ">=2023.5.7" @@ -6492,7 +6670,7 @@ description = "An Amazon S3 Transfer Manager" optional = true python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"snowflake\" or extra == \"all\"" +markers = "extra == \"snowflake\" or extra == \"all\" or extra == \"redshift\"" files = [ {file = "s3transfer-0.18.0-py3-none-any.whl", hash = "sha256:239c13b09e65ad0346e1be7348b8a202dcad44ac7ea7c6eb858fc881dce739b6"}, {file = "s3transfer-0.18.0.tar.gz", hash = "sha256:3760b8b7ec1315da54048b2d626276732bee4300d054d492d4e1d43e20d4ecbd"}, @@ -6504,6 +6682,22 @@ botocore = ">=1.37.4,<2.0a0" [package.extras] crt = ["botocore[crt] (>=1.37.4,<2.0a0)"] +[[package]] +name = "scramp" +version = "1.4.15" +description = "An implementation of the SCRAM protocol." +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "extra == \"redshift\" or extra == \"all\"" +files = [ + {file = "scramp-1.4.15-py3-none-any.whl", hash = "sha256:9d6102948d9005e3802384a328429dfd67d691a65791007c354ff89895857396"}, + {file = "scramp-1.4.15.tar.gz", hash = "sha256:d25cdd3dbc493773647bccb93e4e85bbff0c091141ec8ff8d61bad1e3638082d"}, +] + +[package.dependencies] +asn1crypto = ">=1.5.1" + [[package]] name = "send2trash" version = "2.1.0" @@ -6526,11 +6720,12 @@ version = "82.0.1" description = "Most extensible Python build backend with support for C/C++ extension modules" optional = false python-versions = ">=3.9" -groups = ["dev"] +groups = ["main", "dev"] files = [ {file = "setuptools-82.0.1-py3-none-any.whl", hash = "sha256:a59e362652f08dcd477c78bb6e7bd9d80a7995bc73ce773050228a348ce2e5bb"}, {file = "setuptools-82.0.1.tar.gz", hash = "sha256:7d872682c5d01cfde07da7bccc7b65469d3dca203318515ada1de5eda35efbf9"}, ] +markers = {main = "extra == \"redshift\" or extra == \"all\""} [package.extras] check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1) ; sys_platform != \"cygwin\"", "ruff (>=0.13.0) ; sys_platform != \"cygwin\""] @@ -6700,11 +6895,12 @@ version = "2.8.3" description = "A modern CSS selector implementation for Beautiful Soup." optional = false python-versions = ">=3.9" -groups = ["dev"] +groups = ["main", "dev"] files = [ {file = "soupsieve-2.8.3-py3-none-any.whl", hash = "sha256:ed64f2ba4eebeab06cc4962affce381647455978ffc1e36bb79a545b91f45a95"}, {file = "soupsieve-2.8.3.tar.gz", hash = "sha256:3267f1eeea4251fb42728b6dfb746edc9acaffc4a45b27e19450b676586e8349"}, ] +markers = {main = "extra == \"redshift\" or extra == \"all\""} [[package]] name = "sqlalchemy" @@ -6835,6 +7031,27 @@ bqstorage = ["google-cloud-bigquery-storage (>=2.18.0,<3.0.0)", "grpcio (>=1.47. geography = ["GeoAlchemy2", "shapely"] tests = ["packaging", "pytz"] +[[package]] +name = "sqlalchemy-redshift" +version = "1.0.0" +description = "Amazon Redshift Dialect for sqlalchemy" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "extra == \"redshift\" or extra == \"all\"" +files = [ + {file = "sqlalchemy_redshift-1.0.0-py3-none-any.whl", hash = "sha256:8b3a946401148b9443eb623f84a64bb30fa99ddc873c17d6a9e70cc0e5ea8dd1"}, + {file = "sqlalchemy_redshift-1.0.0.tar.gz", hash = "sha256:882964d42184604ef653ab770501e942e7f9296b537d3b138551f609d1c52860"}, +] + +[package.dependencies] +packaging = "*" +SQLAlchemy = ">=2.0.0,<3" + +[package.extras] +dev = ["black", "isort"] +tests = ["alembic", "psycopg2-binary", "psycopg2cffi", "pytest"] + [[package]] name = "sqlglot" version = "30.11.0" @@ -7413,7 +7630,7 @@ files = [ {file = "urllib3-2.6.3-py3-none-any.whl", hash = "sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4"}, {file = "urllib3-2.6.3.tar.gz", hash = "sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed"}, ] -markers = {main = "(extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"clickhouse\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\") or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\""} +markers = {main = "(extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"redshift\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"clickhouse\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"redshift\") or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"redshift\""} [package.extras] brotli = ["brotli (>=1.2.0) ; platform_python_implementation == \"CPython\"", "brotlicffi (>=1.2.0.0) ; platform_python_implementation != \"CPython\""] @@ -7956,7 +8173,7 @@ files = [ [extras] advanced-search = ["ladybug", "litellm", "numpy"] -all = ["aiomysql", "asyncpg", "clickhouse-sqlalchemy", "dbt-core", "httpx", "ladybug", "litellm", "numpy", "pandas", "psycopg2-binary", "pyarrow", "pymysql", "pyodbc", "snowflake-connector-python", "snowflake-sqlalchemy", "sqlalchemy-bigquery"] +all = ["aiomysql", "asyncpg", "clickhouse-sqlalchemy", "dbt-core", "httpx", "ladybug", "litellm", "numpy", "pandas", "psycopg2-binary", "pyarrow", "pymysql", "pyodbc", "redshift-connector", "snowflake-connector-python", "snowflake-sqlalchemy", "sqlalchemy-bigquery", "sqlalchemy-redshift"] bigquery = ["sqlalchemy-bigquery"] clickhouse = ["clickhouse-sqlalchemy"] client = ["httpx", "pandas"] @@ -7965,10 +8182,11 @@ embedding-search = ["litellm", "numpy"] flight = ["pyarrow"] mysql = ["aiomysql", "pymysql"] postgres = ["asyncpg", "psycopg2-binary"] +redshift = ["redshift-connector", "sqlalchemy-redshift"] snowflake = ["snowflake-connector-python", "snowflake-sqlalchemy"] sqlserver = ["pyodbc"] [metadata] lock-version = "2.1" python-versions = "^3.11" -content-hash = "aa61d7145c9569f852cf60d41cdc64a079de4a68ed23f50afbea30c488b96d17" +content-hash = "68a0abcd21bebda972cd05a26153daff80ee414718c25c7027d9f8a0dd61ff01" diff --git a/pyproject.toml b/pyproject.toml index 9f848e62..85b9b6e5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -79,6 +79,14 @@ ladybug = {version = ">=0.1", optional = true} # release with SQLAlchemy 2.x support (a hard repo dep). snowflake-connector-python = {version = ">=3.13", optional = true} snowflake-sqlalchemy = {version = ">=1.7", optional = true} +# Redshift support. sqlalchemy-redshift has no hard DBAPI dependency of its +# own — it lazily imports whichever driver the connection URL's scheme +# selects. redshift-connector is AWS's actively-maintained driver (IAM auth, +# Redshift Serverless, browser-based SSO); we default the `redshift` type to +# it (see `DatasourceConfig.get_connection_string`) rather than the +# psycopg2-only `redshift://` form. +sqlalchemy-redshift = {version = ">=1.0", optional = true} +redshift-connector = {version = ">=2.1", optional = true} [tool.poetry.extras] client = ["httpx", "pandas"] @@ -88,11 +96,12 @@ clickhouse = ["clickhouse-sqlalchemy"] sqlserver = ["pyodbc"] snowflake = ["snowflake-connector-python", "snowflake-sqlalchemy"] bigquery = ["sqlalchemy-bigquery"] +redshift = ["sqlalchemy-redshift", "redshift-connector"] dbt = ["dbt-core"] flight = ["pyarrow"] advanced_search = ["litellm", "numpy", "ladybug"] embedding_search = ["litellm", "numpy"] # legacy alias for advanced_search without graph support -all = ["httpx", "pandas", "psycopg2-binary", "asyncpg", "pymysql", "aiomysql", "clickhouse-sqlalchemy", "pyodbc", "snowflake-connector-python", "snowflake-sqlalchemy", "sqlalchemy-bigquery", "dbt-core", "pyarrow", "litellm", "numpy", "ladybug"] +all = ["httpx", "pandas", "psycopg2-binary", "asyncpg", "pymysql", "aiomysql", "clickhouse-sqlalchemy", "pyodbc", "snowflake-connector-python", "snowflake-sqlalchemy", "sqlalchemy-bigquery", "sqlalchemy-redshift", "redshift-connector", "dbt-core", "pyarrow", "litellm", "numpy", "ladybug"] [project.scripts] slayer = "slayer.cli:main" diff --git a/slayer/core/models.py b/slayer/core/models.py index 293638d6..a9b9975f 100644 --- a/slayer/core/models.py +++ b/slayer/core/models.py @@ -877,6 +877,18 @@ def get_connection_string(self) -> str: "mysql": "mysql+pymysql", "mariadb": "mysql+pymysql", "clickhouse": "clickhouse+http", + # Explicit rather than falling through to the bare "redshift" + # scheme (which sqlalchemy-redshift also registers, defaulting to + # its psycopg2 dialect): redshift-connector is AWS's maintained + # driver and the one this DBAPI-agnostic sqlalchemy-redshift + # setup is documented against (IAM auth, Redshift Serverless, + # browser SSO — psycopg2 only does password auth). Being + # explicit here also stops a future edit from "simplifying" this + # to "postgresql" — Redshift's wire protocol is Postgres-derived + # but its dialect (RedshiftDialect) carries real SQL differences + # (e.g. APPROXIMATE COUNT(DISTINCT)) that the bare postgresql + # dialect doesn't know about. + "redshift": "redshift+redshift_connector", } driver = driver_map.get(self.type, self.type) # Build the URL with SQLAlchemy's structured builder (issue #240) diff --git a/tests/integration/test_integration_redshift.py b/tests/integration/test_integration_redshift.py new file mode 100644 index 00000000..ef2560cd --- /dev/null +++ b/tests/integration/test_integration_redshift.py @@ -0,0 +1,330 @@ +"""Live integration tests for Redshift. + +Skip-by-default: requires ``sqlalchemy-redshift`` + ``redshift-connector`` +(the ``motley-slayer[redshift]`` extra) and a real Redshift cluster/serverless +endpoint reachable via env vars: + + REDSHIFT_HOST (required — cluster/serverless endpoint, no port) + REDSHIFT_PORT (optional, default 5439) + REDSHIFT_DATABASE (required) + REDSHIFT_USER (required) + REDSHIFT_PASSWORD (required) + +Redshift has no free local Docker image (unlike Postgres/MySQL/ClickHouse/ +SQL Server) and no shared local-profile convention (unlike Snowflake's +``~/.snowflake/connections.toml``), so — like BigQuery — this suite is +env-var-gated rather than file-gated, and skips cleanly in any environment +without those set (including plain `pytest` runs in this repo today). + +Unlike the Snowflake suite, this does NOT assert on auto-ingestion's FK-join +discovery: Redshift allows declaring (unenforced) FOREIGN KEY constraints, +but whether SQLAlchemy's Inspector surfaces them through +``sqlalchemy-redshift`` for auto-ingestion to pick up is unverified absent a +live cluster. Models below define ``joins=`` explicitly instead, which +exercises the query/rollup path regardless of that answer. +""" + +import os +import uuid + +import pytest + +# Skip the entire module if the redshift extras aren't installed. +pytest.importorskip("redshift_connector") +pytest.importorskip("sqlalchemy_redshift") + +import redshift_connector # noqa: E402 + +from slayer.async_utils import run_sync # noqa: E402 +from slayer.core.enums import DataType # noqa: E402 +from slayer.core.models import ( # noqa: E402 + Column, + DatasourceConfig, + ModelJoin, + ModelMeasure, + SlayerModel, +) +from slayer.core.query import ColumnRef, SlayerQuery # noqa: E402 +from slayer.engine.query_engine import SlayerQueryEngine # noqa: E402 +from slayer.storage.yaml_storage import YAMLStorage # noqa: E402 + +_HOST = os.environ.get("REDSHIFT_HOST") +_PORT = int(os.environ.get("REDSHIFT_PORT", "5439")) +_DATABASE = os.environ.get("REDSHIFT_DATABASE") +_USER = os.environ.get("REDSHIFT_USER") +_PASSWORD = os.environ.get("REDSHIFT_PASSWORD") + +pytestmark = [ + pytest.mark.integration, + pytest.mark.skipif( + not (_HOST and _DATABASE and _USER and _PASSWORD), + reason=( + "REDSHIFT_HOST/REDSHIFT_DATABASE/REDSHIFT_USER/REDSHIFT_PASSWORD not all set; skipping Redshift live tests" + ), + ), +] + + +@pytest.fixture(scope="module") +def rs_transient_schema(): + """Module-scoped: a uniquely-named schema created for this test run. + Returns the schema name. Teardown drops it with CASCADE. + + Teardown safety: ``cur`` and ``schema_name`` are bound BEFORE entering + the try block so the finally clause can rely on them existing even when + early setup fails. + """ + conn = redshift_connector.connect( + host=_HOST, + port=_PORT, + database=_DATABASE, + user=_USER, + password=_PASSWORD, + ) + schema_name = f"slayer_test_{uuid.uuid4().hex[:12]}" + cur = conn.cursor() + schema_created = False + try: + cur.execute(f"CREATE SCHEMA {schema_name}") + schema_created = True + cur.execute(f"SET search_path TO {schema_name}") + cur.execute(""" + CREATE TABLE regions ( + id BIGINT PRIMARY KEY, + name VARCHAR(64) NOT NULL + ) + """) + cur.execute(""" + CREATE TABLE customers ( + id BIGINT PRIMARY KEY, + name VARCHAR(128) NOT NULL, + email VARCHAR(256) NOT NULL, + region_id BIGINT + ) + """) + cur.execute(""" + CREATE TABLE products ( + id BIGINT PRIMARY KEY, + name VARCHAR(128) NOT NULL, + category VARCHAR(64) NOT NULL, + price DECIMAL(10, 2) NOT NULL + ) + """) + cur.execute(""" + CREATE TABLE orders ( + id BIGINT PRIMARY KEY, + customer_id BIGINT, + product_id BIGINT, + quantity BIGINT NOT NULL, + status VARCHAR(32) NOT NULL, + created_at TIMESTAMP NOT NULL + ) + """) + cur.executemany( + "INSERT INTO regions VALUES (%s, %s)", + [(1, "US"), (2, "EU"), (3, "APAC")], + ) + cur.executemany( + "INSERT INTO customers VALUES (%s, %s, %s, %s)", + [ + (1, "Acme", "acme@example.com", 1), + (2, "Globex", "globex@example.com", 2), + (3, "Initech", "initech@example.com", 1), + ], + ) + cur.executemany( + "INSERT INTO products VALUES (%s, %s, %s, %s)", + [ + (1, "Widget", "tools", 9.99), + (2, "Gadget", "tools", 19.99), + (3, "Doohickey", "novelty", 4.99), + ], + ) + cur.executemany( + "INSERT INTO orders VALUES (%s, %s, %s, %s, %s, %s)", + [ + (1, 1, 1, 2, "completed", "2024-01-15 10:00:00"), + (2, 1, 2, 1, "completed", "2024-01-20 11:00:00"), + (3, 2, 3, 5, "pending", "2024-02-10 09:00:00"), + (4, 2, 1, 3, "completed", "2024-02-15 14:00:00"), + (5, 3, 2, 1, "cancelled", "2024-03-01 08:00:00"), + (6, 3, 3, 4, "pending", "2024-03-10 16:00:00"), + ], + ) + conn.commit() + yield schema_name + finally: + if schema_created: + try: + cur.execute(f"DROP SCHEMA IF EXISTS {schema_name} CASCADE") + conn.commit() + except Exception: + # Don't mask the original error if setup failed mid-way. + pass + try: + cur.close() + except Exception: + pass + conn.close() + + +@pytest.fixture +def rs_datasource(rs_transient_schema: str) -> DatasourceConfig: + return DatasourceConfig( + name="rs_test", + type="redshift", + host=_HOST, + port=_PORT, + database=_DATABASE, + username=_USER, + password=_PASSWORD, + schema_name=rs_transient_schema, + ) + + +@pytest.fixture +def rs_storage_with_models(rs_datasource: DatasourceConfig, tmp_path): + """YAMLStorage with the four standard models pre-saved, joins declared + explicitly (see module docstring).""" + storage = YAMLStorage(base_dir=str(tmp_path)) + run_sync(storage.save_datasource(rs_datasource)) + + orders = SlayerModel( + name="orders", + sql_table=f"{rs_datasource.schema_name}.orders", + data_source="rs_test", + default_time_dimension="created_at", + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", sql="customer_id", type=DataType.INT), + Column(name="product_id", sql="product_id", type=DataType.INT), + Column(name="quantity", sql="quantity", type=DataType.INT), + Column(name="status", sql="status", type=DataType.TEXT), + Column(name="created_at", sql="created_at", type=DataType.TIMESTAMP), + ], + joins=[ + ModelJoin(target_model="customers", join_pairs=[["customer_id", "id"]]), + ModelJoin(target_model="products", join_pairs=[["product_id", "id"]]), + ], + ) + customers = SlayerModel( + name="customers", + sql_table=f"{rs_datasource.schema_name}.customers", + data_source="rs_test", + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="name", sql="name", type=DataType.TEXT), + Column(name="email", sql="email", type=DataType.TEXT), + Column(name="region_id", sql="region_id", type=DataType.INT), + ], + joins=[ + ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]]), + ], + ) + products = SlayerModel( + name="products", + sql_table=f"{rs_datasource.schema_name}.products", + data_source="rs_test", + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="name", sql="name", type=DataType.TEXT), + Column(name="category", sql="category", type=DataType.TEXT), + Column(name="price", sql="price", type=DataType.DOUBLE), + ], + ) + regions = SlayerModel( + name="regions", + sql_table=f"{rs_datasource.schema_name}.regions", + data_source="rs_test", + columns=[ + Column(name="id", sql="id", type=DataType.INT, primary_key=True), + Column(name="name", sql="name", type=DataType.TEXT), + ], + ) + for m in (orders, customers, products, regions): + run_sync(storage.save_model(m)) + yield storage + + +# --------------------------------------------------------------------------- +# Smoke tests +# --------------------------------------------------------------------------- + + +def test_basic_query(rs_storage_with_models) -> None: + """SELECT sum(quantity) FROM orders — minimum smoke for the engine path, + exercising the redshift-connector driver end to end.""" + engine = SlayerQueryEngine(storage=rs_storage_with_models) + result = run_sync( + engine.execute( + SlayerQuery( + source_model="orders", + measures=[ModelMeasure(formula="quantity:sum")], + ) + ) + ) + rows = result.data + assert len(rows) == 1 + # 6 orders, quantities 2+1+5+3+1+4 = 16 + val = next(iter(rows[0].values())) + assert int(val) == 16 + + +def test_query_with_dimension(rs_storage_with_models) -> None: + """Group-by status; assert 3 status groups.""" + engine = SlayerQueryEngine(storage=rs_storage_with_models) + result = run_sync( + engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count")], + ) + ) + ) + rows = result.data + statuses = {row.get("orders.status") for row in rows} + assert statuses == {"completed", "pending", "cancelled"} + + +def test_rollup_join_via_explicit_joins(rs_storage_with_models) -> None: + """Cross-model measure: orders.customers.regions.name (multi-hop join). + + Seed data: customers 1+3 live in US, customer 2 lives in EU, region 3 + (APAC) has no customers — and the LEFT JOIN chain starts at orders, so + APAC never surfaces. Exercising orders -> customers -> regions correctly + yields exactly {US, EU}.""" + engine = SlayerQueryEngine(storage=rs_storage_with_models) + result = run_sync( + engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="customers.regions.name")], + measures=[ModelMeasure(formula="quantity:sum")], + ) + ) + ) + rows = result.data + region_names = {row.get("orders.customers.regions.name") for row in rows} + assert region_names == {"US", "EU"} + + +def test_approx_count_distinct_uses_approximate_keyword(rs_storage_with_models) -> None: + """RedshiftDialect.build_approx_count_distinct emits the keyword-prefix + form ``APPROXIMATE COUNT(DISTINCT x)`` rather than a native aggregate + function name — this is the one Tier-2 SQL-shape quirk this dialect + encodes (slayer/sql/dialects/_tier2.py). 3 distinct customer_ids across + 6 orders.""" + engine = SlayerQueryEngine(storage=rs_storage_with_models) + query = SlayerQuery( + source_model="orders", + measures=[ModelMeasure(formula="customer_id:count_distinct_approx")], + ) + dry_run_result = run_sync(engine.execute(query, dry_run=True)) + assert "APPROXIMATE COUNT(DISTINCT" in dry_run_result.sql + + result = run_sync(engine.execute(query)) + rows = result.data + assert len(rows) == 1 + val = next(iter(rows[0].values())) + assert int(val) == 3 diff --git a/tests/test_models.py b/tests/test_models.py index 48f4bfac..0672cbd3 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -1328,6 +1328,41 @@ def test_password_without_username_is_dropped(self) -> None: assert url.host == "db.example" assert url.database == "analytics" + def test_redshift_connection_string_uses_redshift_connector_driver(self) -> None: + # Explicit driver_map entry (not the bare "redshift" scheme, which + # sqlalchemy-redshift also registers but defaults to psycopg2 — + # password-auth only). redshift-connector supports IAM auth and + # Redshift Serverless, which is what docs/configuration/datasources.md + # documents as the intended driver. + ds = DatasourceConfig( + name="test", + type="redshift", + host="mycluster.abc123.us-east-1.redshift.amazonaws.com", + port=5439, + database="dev", + username="user", + password="pass", # NOSONAR(S2068) — test-only fixture credential, not a real secret + ) + cs = ds.get_connection_string() + assert cs.startswith("redshift+redshift_connector://") + url = make_url(cs) + assert url.host == "mycluster.abc123.us-east-1.redshift.amazonaws.com" + assert url.port == 5439 + assert url.database == "dev" + assert url.username == "user" + assert url.password == "pass" # NOSONAR(S2068) — test-only fixture credential, not a real secret + + def test_redshift_explicit_connection_string_bypasses_driver_map(self) -> None: + # A caller who wants the psycopg2 dialect explicitly (or the bare + # "redshift" scheme) can still do so via `connection_string` — + # driver_map only applies when building the URL from structured + # fields. + ds = DatasourceConfig( + name="test", + connection_string="redshift+psycopg2://user:pass@host:5439/dev", + ) + assert ds.get_connection_string() == "redshift+psycopg2://user:pass@host:5439/dev" + class TestTimeGranularity: def test_period_start_week(self) -> None: