Add live Redshift connector (driver_map + redshift extra)

.github/workflows/ci.yml

@@ -214,3 +214,53 @@
       - name: Wipe Snowflake credentials
         if: always()
         run: rm -f ~/.snowflake/connections.toml
+
+  redshift-integration:
+    # Skip on forked-PR runs where the Redshift secrets aren't available — the
+    # job would fail otherwise and we'd rather not nag external contributors.
+    if: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }}
+    runs-on: ubuntu-latest
+    needs: lint-and-test
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Skip if Redshift secrets are unset
+        id: gate
+        env:
+          REDSHIFT_HOST: ${{ secrets.REDSHIFT_HOST }}
+        run: |
+          if [[ -z "$REDSHIFT_HOST" ]]; then
+            echo "Redshift secrets unset — skipping the integration suite."
+            echo "skip=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "skip=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Set up Python
+        if: steps.gate.outputs.skip != 'true'
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install Poetry
+        if: steps.gate.outputs.skip != 'true'
+        run: pip install poetry
+
+      - name: Install dependencies (with redshift extra)
+        if: steps.gate.outputs.skip != 'true'
+        run: poetry install -E redshift
+
+      - name: Run Redshift integration tests
+        if: steps.gate.outputs.skip != 'true'
+        timeout-minutes: 15
+        # Points at a persistent dev cluster/serverless endpoint (unlike the
+        # Docker-backed suites, there's no ephemeral-container option for
+        # Redshift) — tests create and drop their own uniquely-named schema,
+        # see tests/integration/test_integration_redshift.py.
+        env:
+          REDSHIFT_HOST: ${{ secrets.REDSHIFT_HOST }}
+          REDSHIFT_PORT: ${{ secrets.REDSHIFT_PORT }}
+          REDSHIFT_DATABASE: ${{ secrets.REDSHIFT_DATABASE }}
+          REDSHIFT_USER: ${{ secrets.REDSHIFT_USER }}
+          REDSHIFT_PASSWORD: ${{ secrets.REDSHIFT_PASSWORD }}
+        run: poetry run pytest tests/integration/test_integration_redshift.py -v -m integration --timeout=300

DECISIONS.md

@@ -69,3 +69,4 @@ implementation detail. Include issue refs when known.
 - 2026-08-03 — Optional blocks + Cube JS/FILTER_PARAMS import (DEV-1730 / #270): a Mode-A-only `{? ... ?}` block renders its content parenthesised when every inner `{var}` is supplied, else collapses to the neutral `(1=1)` — the SLayer form of a Cube `FILTER_PARAMS` optional pushdown. Blocks live in the same `substitute_variables` (escape="sql") scanner as `{var}`/`{{`/`}}`, must contain ≥1 var, do not nest, and are rejected in Mode-B. A block-bearing model runs substitution even on a zero-variable call so its blocks collapse (the `_substitute_model_sql_surfaces` fast-path now checks for `{?` too); a block-free, required-only model with zero variables is still left untouched (the documented DEV-1625 raw-brace-literal boundary). `extract_model_variables(model)` derives required (bare, no default) vs optional (in-block or defaulted) from the four Mode-A surfaces — structural, nothing persisted, surfaced additively in the inspect skeleton `Variables:` line. The Cube importer gains a **JavaScript front-end** (esprima ESTree parser, a new core dep) that parses `cube()`/`view()` into the same `CubeCube`/`CubeView` shapes as YAML (dynamic values → report + skip member). FILTER_PARAMS refs are carried JS→converter as structured `CubeFilterParamRef` on the transient `CubeCube` (sentinels in the surface text; no arrow-body re-parse, sidestepping the `{var}`-vs-`{FILTER_PARAMS…}` brace clash); the converter resolves sentinels AFTER `translate_cube_refs` so the introduced `{var}` are never eaten. Requiredness (bare vs block) is decided in the converter alone via `honor_required_meta` (default on; CLI `--ignore-required-meta`) AND the member's `meta.required`; with the flag off a scalar-position arrow collapses to Cube's own `(1=1)::TIMESTAMP` booby-trap, faithfully. Cross-cube refs, unknown members, and generated-name collisions (`d`→`d_from` clashing member `d_from`) drop the cube (`filter_params_unsupported`); each logical variable is reported once (`filter_params_variable`) and stashed in `meta.cube_variables`. `render_probe_text` (blocks→`(1=1)`, bare vars→`0`) is the single import-time validation renderer, matching runtime collapse.
 - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678.
 - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection.
+- 2026-08-05 — Redshift live connector (`redshift-live-connector` branch): manual pass against a personal Redshift Serverless endpoint does not promote Redshift from Tier 2 to Tier 1 — the tier's meaning is CI-enforced non-regression, and no CI job runs it yet. Deliberately not wiring `REDSHIFT_*` secrets into `redshift-integration` CI until a project-owned (non-personal) endpoint exists: the job's fork-PR skip does not gate same-repo pushes/PRs, so pointing it at a personal AWS resource would bill every contributor's CI run to that account and expose the credential to anything running in that job.

docs/configuration/datasources.md

@@ -64,12 +64,12 @@ These databases are verified by integration tests and runnable Docker examples.
 
 #### Additional support
 
-SQL generation is covered by unit tests, but not verified against live instances. Install the appropriate SQLAlchemy driver manually.
+SQL generation is covered by unit tests, but not verified against live instances.
 
 | Type | SQLAlchemy Driver | Install |
 |------|-------------------|---------|
 | `bigquery` | `sqlalchemy-bigquery` | `pip install sqlalchemy-bigquery` |
-| `redshift` | `sqlalchemy-redshift` + `redshift_connector` | `pip install sqlalchemy-redshift redshift-connector` |
+| `redshift` | `sqlalchemy-redshift` + `redshift-connector` | `pip install 'motley-slayer[redshift]'`. Connection-layer code is unit-tested and has passed manually against a live Redshift Serverless endpoint (outside CI) — see [Redshift](#redshift) below. |
 | `trino` / `presto` / `athena` | `trino` or `PyAthena` | `pip install trino` or `pip install PyAthena` |
 | `databricks` / `spark` | `databricks-sql-connector` | `pip install databricks-sql-connector` |
 | `oracle` | `oracledb` | `pip install oracledb` |
@@ -148,6 +148,57 @@ Statement-level timeout is enforced via
 !!! tip
     If your database isn't listed but is supported by sqlglot, it may already work — SLayer falls back to Postgres-style SQL by default. Try it and [open an issue](https://github.com/MotleyAI/slayer/issues) if you hit a problem.
 
+### Redshift
+
+```bash
+pip install 'motley-slayer[redshift]'
+```
+
+The extra pulls in `sqlalchemy-redshift` + `redshift-connector`. Structured
+`DatasourceConfig` fields (`host`, `port`, `database`, `username`, `password`)
+build a `redshift+redshift_connector://` URL by default:
+
+```yaml
+# datasources/rs.yaml
+name: rs
+type: redshift
+host: mycluster.abc123.us-east-1.redshift.amazonaws.com
+port: 5439
+database: dev
+username: YOUR_USER
+password: YOUR_PASSWORD
+```
+
+`redshift-connector` is AWS's actively-maintained driver — it supports IAM
+auth, Redshift Serverless, and browser-based SSO in addition to plain
+username/password. To use the classic psycopg2-based dialect instead (e.g.
+for parity with an existing psycopg2 connection pool), set `connection_string`
+directly rather than the structured fields. This requires `psycopg2-binary`,
+which the `redshift` extra does not install — pull it in via the `postgres`
+extra or install it separately:
+
+```bash
+pip install 'motley-slayer[redshift,postgres]'
+```
+
+```yaml
+name: rs
+connection_string: "redshift+psycopg2://user:pass@host:5439/dev"
+```
+
+!!! warning "No native FK-based auto-ingestion join discovery (unverified)"
+    Redshift allows declaring (unenforced) `FOREIGN KEY` constraints, but
+    whether `sqlalchemy-redshift`'s Inspector surfaces them for auto-ingestion
+    to discover joins from — the way it does for Postgres / MySQL / SQLite /
+    Snowflake — has not been verified against a live cluster. Until confirmed,
+    treat Redshift like BigQuery/ClickHouse: define `joins:` manually in your
+    model YAML rather than relying on auto-discovery.
+
+!!! note "`APPROXIMATE COUNT(DISTINCT x)`"
+    Redshift's `count_distinct_approx` aggregation uses the keyword-prefix
+    form `APPROXIMATE COUNT(DISTINCT x)` rather than a native aggregate
+    function name (see [Aggregation support](../database-support.md)).
+
 ## Field Reference
 
 | Field | Type | Required | Description |

docs/database-support.md

@@ -32,8 +32,23 @@ testcontainers suites.
 
 Unit tests for SQL generation; no live-instance verification.
 
-Redshift, Trino/Presto (Athena uses the Presto dialect), Databricks/Spark,
-Oracle.
+Trino/Presto (Athena uses the Presto dialect), Databricks/Spark, Oracle.
+
+**Redshift** is further along than the rest of this tier: `RedshiftDialect`
+(SQL generation) has been code-covered here for a while, and the connection
+layer — `driver_map["redshift"] = "redshift+redshift_connector"`, the
+`motley-slayer[redshift]` extra, and a live integration suite
+(`tests/integration/test_integration_redshift.py`, gated on `REDSHIFT_HOST` /
+`REDSHIFT_DATABASE` / `REDSHIFT_USER` / `REDSHIFT_PASSWORD`) — now exist too.
+The suite has passed manually against a live Redshift Serverless endpoint
+(outside CI). It stays in Tier 2 rather than Tier 1 because the
+`redshift-integration` CI job (`.github/workflows/ci.yml`) has not yet run it
+against a live endpoint — CI secrets are deliberately not wired up yet, since
+the only endpoint verified so far is a personal AWS resource, and pointing CI
+at it would bill every contributor's CI run to that personal account. This
+entry should move to Tier 1 once the CI job passes at least once against a
+project-owned (non-personal) Redshift endpoint, the same skip-until-configured
+way Snowflake/BigQuery started.
 
 ## Aggregation support
 

examples/redshift/README.md

@@ -0,0 +1,84 @@
+# Redshift example
+
+Redshift is a Tier 2 dialect (see [database support](../../docs/database-support.md)):
+`RedshiftDialect`'s SQL generation is unit-tested, and the connection layer added
+alongside this example (`motley-slayer[redshift]`, `driver_map["redshift"]`) is
+too. The connection layer has since passed manually against a live Redshift
+Serverless endpoint (outside CI) — see the CI note under Known limitations.
+Like Snowflake / BigQuery, there's no free local Docker image, so this example
+needs a real cluster or Redshift Serverless workgroup.
+
+## 1. Install the extra
+
+```bash
+pip install 'motley-slayer[redshift]'
+```
+
+The extra pulls in `sqlalchemy-redshift` + `redshift-connector`.
+
+## 2. Configure a connection
+
+```yaml
+# slayer_data/datasources/rs.yaml
+name: rs
+type: redshift
+host: mycluster.abc123.us-east-1.redshift.amazonaws.com
+port: 5439
+database: dev
+username: YOUR_USER
+password: YOUR_PASSWORD
+```
+
+Or as a connection string, using `redshift-connector` (the default driver —
+supports IAM auth and Redshift Serverless, not just password auth):
+
+```text
+redshift+redshift_connector://YOUR_USER:YOUR_PASSWORD@mycluster.abc123.us-east-1.redshift.amazonaws.com:5439/dev
+```
+
+## 3. Seed the demo schema
+
+```bash
+python ../seed.py "redshift+redshift_connector://YOUR_USER:YOUR_PASSWORD@mycluster.abc123.us-east-1.redshift.amazonaws.com:5439/dev"
+```
+
+This drops + recreates the four canonical tables (`regions`, `customers`,
+`products`, `orders`) and inserts the standard fixture dataset.
+
+## 4. Register the datasource and ingest
+
+```bash
+slayer datasources create "redshift+redshift_connector://YOUR_USER:YOUR_PASSWORD@.../dev" --name rs --ingest
+```
+
+**Redshift allows declaring `FOREIGN KEY` constraints, but they're not
+enforced, and whether they're reflected for auto-ingestion's join discovery
+is unverified against a live cluster.** Treat this like the ClickHouse /
+BigQuery examples rather than the Postgres / MySQL / Snowflake ones: assume
+joins are **not** auto-generated, and add them manually to the model YAML
+files under `slayer_data/models/` if you need cross-model rollups.
+
+## 5. Verify
+
+```bash
+python verify.py
+```
+
+`verify.py` runs the same battery used by the other examples: schema/type
+checks, aggregation matrix (`median`, `percentile`, `stddev_samp/pop`,
+`var_samp/pop`, `corr`, `covar_samp/pop`), and `count_distinct_approx` — which
+on Redshift compiles to the keyword-prefix form `APPROXIMATE COUNT(DISTINCT x)`
+rather than a native aggregate function name.
+
+## Known limitations
+
+- **No auto-discovered joins** (see step 4) — unverified either way; add
+  `joins:` manually until confirmed.
+- **Not yet CI-verified.** `tests/integration/test_integration_redshift.py`
+  has passed manually against a live Redshift Serverless endpoint, but the
+  `redshift-integration` CI job still skips — `REDSHIFT_HOST` /
+  `REDSHIFT_DATABASE` / `REDSHIFT_USER` / `REDSHIFT_PASSWORD` repo secrets are
+  deliberately not configured yet, since the only endpoint verified so far is
+  a personal one and wiring it into CI would bill every contributor's run to
+  that account. Redshift moves to Tier 1 once the job passes against a
+  project-owned endpoint.