diff --git a/docs-mintlify/docs/explore-analyze/charts/configuration/small-multiples.mdx b/docs-mintlify/docs/explore-analyze/charts/configuration/small-multiples.mdx
index 62bb912cd4bbe..71b2a4ee07ada 100644
--- a/docs-mintlify/docs/explore-analyze/charts/configuration/small-multiples.mdx
+++ b/docs-mintlify/docs/explore-analyze/charts/configuration/small-multiples.mdx
@@ -23,7 +23,7 @@ These options appear once a **Split by** dimension is chosen.
| Option | What it does |
|---|---|
-| **Grid** | Columns × rows, up to 5 × 4. Both are preselected from the number of distinct values in the split dimension, so a four-value dimension opens as a 2 × 2 grid. |
+| **Grid** | Columns × rows, up to 5 × 5. Both are preselected from the number of distinct values in the split dimension, so a four-value dimension opens as a 2 × 2 grid. |
| **Axis scales** | Whether every panel is drawn against the same scale (**Shared**) or each scales to its own data (**Independent**). |
| **Sort panels by** | Orders the panels by the dimension's own values (**Value**) or by a measure (**Measure**). |
| **Sort order** | **Ascending** or **Descending**. |
@@ -38,7 +38,7 @@ Switch to **Independent** when the question is about the *shape* of each series
### How many panels are drawn
-The grid bounds the render: a chart split into 3 × 2 draws at most six panels, so a high-cardinality dimension can never produce hundreds of unreadable ones. The largest grid is 5 × 4, or twenty panels.
+The grid bounds the render: a chart split into 3 × 2 draws at most six panels, so a high-cardinality dimension can never produce hundreds of unreadable ones. The largest grid is 5 × 5, or twenty-five panels.
When a dimension has more values than the grid has tiles, the chart draws the first ones in the current sort order. The underlying query is unaffected, so no data is lost — a bigger grid simply shows more of it.
diff --git a/docs-mintlify/embedding/multitenancy.mdx b/docs-mintlify/embedding/multitenancy.mdx
index 64c9771f7ed20..96c9424c219e1 100644
--- a/docs-mintlify/embedding/multitenancy.mdx
+++ b/docs-mintlify/embedding/multitenancy.mdx
@@ -199,9 +199,10 @@ cube(`products`, {
### Running in Production
-Each unique id generated by `contextToAppId` or `contextToOrchestratorId` will
-generate a dedicated set of resources, including data model compile cache, SQL
-compile cache, query queues, in-memory result caching, etc. Depending on your
+Each unique id generated by `contextToAppId` will generate a dedicated data model
+compile cache and SQL compile cache, and each unique id generated by
+`contextToOrchestratorId` will generate a dedicated query orchestrator with its
+own database connections, query queues, and in-memory result cache. Depending on your
data model complexity and usage patterns, those resources can have a pretty
sizable memory footprint ranging from single-digit MBs on the lower end and
dozens of MBs on the higher end. So you should make sure Node VM has enough
@@ -250,6 +251,40 @@ module.exports = {
}
```
+## Same DB Instance with per Tenant Data Model
+
+If [`context_to_app_id`][ref-config-ctx-to-appid] is the only option you define,
+Cube compiles and caches a separate data model for each app id it returns. This
+is what enables [`COMPILE_CONTEXT`][ref-cube-security-ctx] and per-tenant
+[`repository_factory`][ref-config-repofactory], so each tenant can get a
+different data model built from the same or different data model files.
+
+**cube.js:**
+
+```javascript
+module.exports = {
+ contextToAppId: ({ securityContext }) =>
+ `CUBE_APP_${securityContext.tenantId}`
+}
+```
+
+
+
+`context_to_app_id` scopes multitenancy to data model compilation only.
+Everything below the data model remains shared across all tenants: database
+connections, query queues, the in-memory query results cache, and
+pre-aggregations, including their tables in the pre-aggregations schema. So
+tenants whose data models generate the same SQL will share queue entries, cached
+results, and pre-aggregation tables.
+
+To isolate those resources per tenant as well, add
+[`pre_aggregations_schema`][ref-config-preagg-schema] for per-tenant
+pre-aggregation tables and [`context_to_orchestrator_id`][ref-config-ctx-to-orch-id]
+for per-tenant connections, queues, and results cache, as shown in the sections
+below.
+
+
+
## Multiple DB Instances with Same Data Model
Let's consider an example where we store data for different users in different
@@ -280,10 +315,11 @@ select the database, based on the `appId` and `userId`:
The App ID (the result of [`contextToAppId`][ref-config-ctx-to-appid]) is used
-as a caching key for various in-memory structures like data model compilation
-results, connection pool. The Orchestrator ID (the result of
+as a caching key for the compiled data model and other data model-related
+in-memory structures. The Orchestrator ID (the result of
[`contextToOrchestratorId`][ref-config-ctx-to-orch-id]) is used as a caching key
-for database connections, execution queues and pre-aggregation table caches. Not
+for the query orchestrator, which holds database connections and their pools,
+execution queues, query results cache, and pre-aggregation table caches. Not
declaring these properties will result in unexpected caching issues such as the
data model or data of one tenant being used for another.
diff --git a/docs-mintlify/recipes/configuration/custom-data-model-per-tenant.mdx b/docs-mintlify/recipes/configuration/custom-data-model-per-tenant.mdx
index 2b3bc5d01ab91..ee407f38d88bc 100644
--- a/docs-mintlify/recipes/configuration/custom-data-model-per-tenant.mdx
+++ b/docs-mintlify/recipes/configuration/custom-data-model-per-tenant.mdx
@@ -23,6 +23,16 @@ from security contexts.
- [`scheduled_refresh_contexts`][ref-scheduled-refresh-contexts] to provide
a list of security contexts.
+
+
+`context_to_app_id` makes the data model per-tenant, which is all this recipe
+needs. Querying and caching (database connections, query queues, and
+pre-aggregations) stay shared between tenants unless you also configure
+[`context_to_orchestrator_id`][ref-context-to-orchestrator-id] and
+[`pre_aggregations_schema`][ref-pre-aggregations-schema].
+
+
+
Put the following code into your `cube.py` or `cube.js` [configuration
file][ref-config-files]:
@@ -525,6 +535,8 @@ code that fetches data model files for each tenant.
[ref-multitenancy]: /embedding/multitenancy
[ref-scheduled-refresh-contexts]: /reference/configuration/config#scheduled_refresh_contexts
[ref-context-to-app-id]: /reference/configuration/config#context_to_app_id
+[ref-context-to-orchestrator-id]: /reference/configuration/config#context_to_orchestrator_id
+[ref-pre-aggregations-schema]: /reference/configuration/config#pre_aggregations_schema
[ref-config-files]: /admin/connect-to-data#cubepy-and-cubejs-files
[ref-mls]: /docs/data-modeling/access-control/member-level-security
[ref-cubes-public]: /reference/data-modeling/cube#public
diff --git a/docs-mintlify/recipes/configuration/multiple-sources-same-schema.mdx b/docs-mintlify/recipes/configuration/multiple-sources-same-schema.mdx
index c789b2d60b79c..1bd1cd550c19d 100644
--- a/docs-mintlify/recipes/configuration/multiple-sources-same-schema.mdx
+++ b/docs-mintlify/recipes/configuration/multiple-sources-same-schema.mdx
@@ -18,7 +18,11 @@ same data model.
To enable multitenancy, use the
[`contextToAppId`](/reference/configuration/config#context_to_app_id) function to
-provide distinct identifiers for each tenant. Also, implement the
+provide distinct identifiers for each tenant's data model and the
+[`contextToOrchestratorId`](/reference/configuration/config#context_to_orchestrator_id)
+function to give each tenant its own database connections and query queues.
+Without the latter, all tenants would share a single connection to whichever
+data source was resolved first. Also, implement the
[`driverFactory`](/reference/configuration/config#driver_factory) function where
you can select a data source based on the tenant name.
[JSON Web Token](/docs/data-modeling/access-control) includes information about the tenant name in
@@ -26,10 +30,15 @@ the `tenant` property of the `securityContext`.
```javascript
module.exports = {
- // Provides distinct identifiers for each tenant which are used as caching keys
+ // Provides a distinct identifier for each tenant's compiled data model
contextToAppId: ({ securityContext }) =>
`CUBE_APP_${securityContext.tenant}`,
+ // Provides a distinct identifier for each tenant's query orchestrator, so
+ // that every tenant gets its own database connections and query queues
+ contextToOrchestratorId: ({ securityContext }) =>
+ `CUBE_APP_${securityContext.tenant}`,
+
// Selects the database connection configuration based on the tenant name
driverFactory: ({ securityContext }) => {
if (!securityContext.tenant) {
diff --git a/docs-mintlify/reference/configuration/config.mdx b/docs-mintlify/reference/configuration/config.mdx
index f0282c343ba0b..db729738c9849 100644
--- a/docs-mintlify/reference/configuration/config.mdx
+++ b/docs-mintlify/reference/configuration/config.mdx
@@ -51,9 +51,27 @@ or when a more flexible setup is needed.
It's a [multitenancy][ref-multitenancy] option.
-`context_to_app_id` is a function to determine an app id which is used as
-caching key for various in-memory structures like data model compilation
-results, etc.
+`context_to_app_id` is a function to determine an app id which is used as the
+caching key for the compiled data model and other data model-related in-memory
+structures. Each distinct app id gets its own data model compilation, which is
+what enables [`COMPILE_CONTEXT`][ref-compile-context] and per-tenant
+[`repository_factory`][self-repofactory].
+
+
+
+`context_to_app_id` enables multitenancy for data model compilation only. On its
+own, it leaves querying and caching shared across app ids. To isolate those per
+tenant as well:
+
+- [`pre_aggregations_schema`][self-pre-aggregations-schema] gives each tenant its
+ own pre-aggregation tables. It is resolved per app id, so a schema name derived
+ from the security context requires `context_to_app_id` to be defined as well.
+- [`context_to_orchestrator_id`][self-orchestrator-id] gives each tenant its own
+ database connections, query queues, and query results cache.
+
+See [multitenancy][ref-multitenancy] for details.
+
+
Called on each request.
@@ -633,6 +651,16 @@ be used exclusively by Cube and shall not be shared with any application.
Called once per [`app_id`][self-opts-ctx-to-appid].
+
+
+Because this option is evaluated once per app id, deriving the schema name from
+the security context requires [`context_to_app_id`][self-opts-ctx-to-appid] to be
+defined as well. Otherwise all tenants share a single app id, and every one of
+them ends up pinned to the schema name resolved for whichever tenant triggered
+the first data model compilation.
+
+
+
```python title="Python"
@@ -1527,6 +1555,7 @@ module.exports = {
[ref-rest-scopes]: /reference/core-data-apis/rest-api#api-scopes
[ref-config-options]: /admin/connect-to-data#configuration-options
[self-orchestrator-id]: #context_to_orchestrator_id
+[ref-compile-context]: /reference/data-modeling/context-variables#compile_context
[ref-multiple-data-sources]: /admin/connect-to-data/multiple-data-sources
[ref-websockets]: /recipes/core-data-api/real-time-data-fetch
[ref-matching-preaggs]: /docs/pre-aggregations/matching-pre-aggregations
diff --git a/docs-mintlify/reference/core-data-apis/sql-api/security.mdx b/docs-mintlify/reference/core-data-apis/sql-api/security.mdx
index cb2d21f3d24e0..c229c12815168 100644
--- a/docs-mintlify/reference/core-data-apis/sql-api/security.mdx
+++ b/docs-mintlify/reference/core-data-apis/sql-api/security.mdx
@@ -51,8 +51,8 @@ configuration file:
```javascripttitle="cube.js"
module.exports = {
- // Create a new appId for each team, this prevents teams from seeing each
- // other's data
+ // Create a new appId for each team so that a separate data model is
+ // compiled for each of them and `COMPILE_CONTEXT` can be used below
// https://cube.dev/docs/product/configuration/reference/config#context_to_app_id
contextToAppId: ({ securityContext }) => {
return securityContext.team
diff --git a/examples/recipes/multiple-data-sources/cube.js b/examples/recipes/multiple-data-sources/cube.js
index dd87548280244..abbc17ed011cf 100644
--- a/examples/recipes/multiple-data-sources/cube.js
+++ b/examples/recipes/multiple-data-sources/cube.js
@@ -1,10 +1,17 @@
const PostgresDriver = require('@cubejs-backend/postgres-driver');
module.exports = {
- // Provides distinct identifiers for each tenant which are used as caching keys
+ // Provides a distinct identifier for each tenant's compiled data model
contextToAppId: ({ securityContext }) =>
`CUBEJS_APP_${securityContext.tenant}`,
+ // Provides a distinct identifier for each tenant's query orchestrator, so
+ // that every tenant gets its own database connections and query queues.
+ // Without this, all tenants would share the driver created for whichever
+ // tenant connected first.
+ contextToOrchestratorId: ({ securityContext }) =>
+ `CUBEJS_APP_${securityContext.tenant}`,
+
// Selects the database connection configuration based on the tenant name
driverFactory: ({ securityContext }) => {
diff --git a/rust/cubesql/cubesql/src/compile/test/snapshots/cubesql__compile__test__test_introspection__domo_column_inspection_query.snap b/rust/cubesql/cubesql/src/compile/test/snapshots/cubesql__compile__test__test_introspection__domo_column_inspection_query.snap
new file mode 100644
index 0000000000000..1faa6f969e7df
--- /dev/null
+++ b/rust/cubesql/cubesql/src/compile/test/snapshots/cubesql__compile__test__test_introspection__domo_column_inspection_query.snap
@@ -0,0 +1,25 @@
+---
+source: cubesql/src/compile/test/test_introspection.rs
+expression: "execute_query(r#\"\n SELECT\n a.attname AS column_name,\n format_type(a.atttypid, NULL) AS data_type,\n a.atttypid = ANY ('{int8,numeric,bool}'::regtype[]) AS is_matched,\n a.atttypid::regtype = ANY ('{int8,numeric,bool}'::regtype[]) AS is_matched_as_regtype,\n CASE\n WHEN a.atttypid = ANY ('{int,int8,int2}'::regtype[]) THEN NULL\n ELSE CASE\n WHEN a.atttypmod = -1 THEN NULL\n ELSE (a.atttypmod - 4) >> 16\n END\n END AS numeric_precision,\n CASE\n WHEN a.atttypid = ANY ('{int,int8,int2}'::regtype[]) THEN NULL\n ELSE CASE\n WHEN a.atttypmod = -1 THEN NULL\n ELSE (a.atttypmod - 4) & 65535\n END\n END AS numeric_scale\n FROM pg_catalog.pg_attribute a\n JOIN pg_catalog.pg_class c ON a.attrelid = c.oid\n JOIN pg_catalog.pg_namespace n ON c.relnamespace = n.oid\n WHERE c.relname = 'KibanaSampleDataEcommerce'\n AND n.nspname = 'public'\n AND a.attnum > 0\n AND NOT a.attisdropped\n AND c.relkind IN ('r', 'v', 'm')\n ORDER BY a.attnum\n \"#.to_string(),\nDatabaseProtocol::PostgreSQL).await?"
+---
++--------------------+-----------------------------+------------+-----------------------+-------------------+---------------+
+| column_name | data_type | is_matched | is_matched_as_regtype | numeric_precision | numeric_scale |
++--------------------+-----------------------------+------------+-----------------------+-------------------+---------------+
+| count | bigint | true | true | NULL | NULL |
+| maxPrice | numeric | true | true | NULL | NULL |
+| sumPrice | numeric | true | true | NULL | NULL |
+| minPrice | numeric | true | true | NULL | NULL |
+| avgPrice | numeric | true | true | NULL | NULL |
+| countDistinct | bigint | true | true | NULL | NULL |
+| id | numeric | true | true | NULL | NULL |
+| order_date | timestamp without time zone | false | false | NULL | NULL |
+| last_mod | timestamp without time zone | false | false | NULL | NULL |
+| customer_gender | text | false | false | NULL | NULL |
+| notes | text | false | false | NULL | NULL |
+| taxful_total_price | numeric | true | true | NULL | NULL |
+| has_subscription | boolean | true | true | NULL | NULL |
+| is_male | boolean | true | true | NULL | NULL |
+| is_female | boolean | true | true | NULL | NULL |
+| __user | text | false | false | NULL | NULL |
+| __cubeJoinField | text | false | false | NULL | NULL |
++--------------------+-----------------------------+------------+-----------------------+-------------------+---------------+
diff --git a/rust/cubesql/cubesql/src/compile/test/snapshots/cubesql__compile__test__test_introspection__empty_regtype_array_comparison.snap b/rust/cubesql/cubesql/src/compile/test/snapshots/cubesql__compile__test__test_introspection__empty_regtype_array_comparison.snap
new file mode 100644
index 0000000000000..9c3b1cbffa8bd
--- /dev/null
+++ b/rust/cubesql/cubesql/src/compile/test/snapshots/cubesql__compile__test__test_introspection__empty_regtype_array_comparison.snap
@@ -0,0 +1,9 @@
+---
+source: cubesql/src/compile/test/test_introspection.rs
+expression: "execute_query(r#\"\n SELECT\n a.atttypid = ANY ('{}'::regtype[]) AS is_matched,\n a.atttypid::regtype = ANY ('{}'::regtype[]) AS is_matched_as_regtype,\n a.atttypid::regtype = ANY (ARRAY[]) AS is_matched_in_untyped_array\n FROM pg_catalog.pg_attribute a\n JOIN pg_catalog.pg_class c ON a.attrelid = c.oid\n WHERE c.relname = 'KibanaSampleDataEcommerce'\n AND a.attnum = 1\n \"#.to_string(),\nDatabaseProtocol::PostgreSQL).await?"
+---
++------------+-----------------------+-----------------------------+
+| is_matched | is_matched_as_regtype | is_matched_in_untyped_array |
++------------+-----------------------+-----------------------------+
+| false | false | false |
++------------+-----------------------+-----------------------------+
diff --git a/rust/cubesql/cubesql/src/compile/test/test_introspection.rs b/rust/cubesql/cubesql/src/compile/test/test_introspection.rs
index b6546bb19ee79..de2a53a276560 100644
--- a/rust/cubesql/cubesql/src/compile/test/test_introspection.rs
+++ b/rust/cubesql/cubesql/src/compile/test/test_introspection.rs
@@ -3437,3 +3437,81 @@ async fn test_pg_user_mapping() -> Result<(), CubeError> {
Ok(())
}
+
+/// Domo's live Postgres connection issues this to inspect the columns of a table picked in
+/// its UI. It tests `atttypid` against a set of types spelled as a `regtype[]` literal.
+///
+/// The `is_matched` columns are not part of Domo's query; they project the type test itself
+/// so that the snapshot pins which OIDs a `regtype[]` literal resolves to. Every row here has
+/// `atttypmod = -1`, so Domo's own columns are `NULL` either way.
+#[tokio::test]
+async fn domo_column_inspection_query() -> Result<(), CubeError> {
+ insta::assert_snapshot!(
+ "domo_column_inspection_query",
+ execute_query(
+ r#"
+ SELECT
+ a.attname AS column_name,
+ format_type(a.atttypid, NULL) AS data_type,
+ a.atttypid = ANY ('{int8,numeric,bool}'::regtype[]) AS is_matched,
+ a.atttypid::regtype = ANY ('{int8,numeric,bool}'::regtype[]) AS is_matched_as_regtype,
+ CASE
+ WHEN a.atttypid = ANY ('{int,int8,int2}'::regtype[]) THEN NULL
+ ELSE CASE
+ WHEN a.atttypmod = -1 THEN NULL
+ ELSE (a.atttypmod - 4) >> 16
+ END
+ END AS numeric_precision,
+ CASE
+ WHEN a.atttypid = ANY ('{int,int8,int2}'::regtype[]) THEN NULL
+ ELSE CASE
+ WHEN a.atttypmod = -1 THEN NULL
+ ELSE (a.atttypmod - 4) & 65535
+ END
+ END AS numeric_scale
+ FROM pg_catalog.pg_attribute a
+ JOIN pg_catalog.pg_class c ON a.attrelid = c.oid
+ JOIN pg_catalog.pg_namespace n ON c.relnamespace = n.oid
+ WHERE c.relname = 'KibanaSampleDataEcommerce'
+ AND n.nspname = 'public'
+ AND a.attnum > 0
+ AND NOT a.attisdropped
+ AND c.relkind IN ('r', 'v', 'm')
+ ORDER BY a.attnum
+ "#
+ .to_string(),
+ DatabaseProtocol::PostgreSQL
+ )
+ .await?
+ );
+
+ Ok(())
+}
+
+/// An empty `regtype[]` literal is rewritten to an empty `ARRAY[]`, which has to plan and
+/// compare false, as Postgres answers for an empty array. Pins both spellings: the column keeps
+/// its OID when the literal resolves, and renders as a type name when there is no regtype to
+/// key off, as with the untyped `ARRAY[]`.
+#[tokio::test]
+async fn empty_regtype_array_comparison() -> Result<(), CubeError> {
+ insta::assert_snapshot!(
+ "empty_regtype_array_comparison",
+ execute_query(
+ r#"
+ SELECT
+ a.atttypid = ANY ('{}'::regtype[]) AS is_matched,
+ a.atttypid::regtype = ANY ('{}'::regtype[]) AS is_matched_as_regtype,
+ a.atttypid::regtype = ANY (ARRAY[]) AS is_matched_in_untyped_array
+ FROM pg_catalog.pg_attribute a
+ JOIN pg_catalog.pg_class c ON a.attrelid = c.oid
+ WHERE c.relname = 'KibanaSampleDataEcommerce'
+ AND a.attnum = 1
+ "#
+ .to_string(),
+ DatabaseProtocol::PostgreSQL
+ )
+ .await?
+ );
+
+ Ok(())
+}
diff --git a/rust/cubesql/cubesql/src/sql/statement.rs b/rust/cubesql/cubesql/src/sql/statement.rs
index af3b33164f3c3..3a301f5379032 100644
--- a/rust/cubesql/cubesql/src/sql/statement.rs
+++ b/rust/cubesql/cubesql/src/sql/statement.rs
@@ -854,9 +854,362 @@ impl CastReplacer {
_ => None,
}
}
+
+ /// True for the `regtype` OID alias type, with or without a `pg_catalog.` qualifier.
+ fn is_regtype(data_type: &ast::DataType) -> bool {
+ match data_type {
+ ast::DataType::Custom(name, _) => {
+ let name = name.to_string().to_lowercase();
+ let name = name.strip_prefix("pg_catalog.").unwrap_or(&name);
+
+ name == "regtype"
+ }
+ _ => false,
+ }
+ }
+
+ /// Resolves a Postgres type name as accepted on `regtype` input to its `pg_type.oid`.
+ /// Understands `pg_type.typname` (`int4`), the canonical regtype spelling (`integer`),
+ /// SQL standard aliases (`int`), a `pg_catalog.` qualifier, a trailing type modifier and a
+ /// trailing `[]`. A modifier is discarded, as Postgres does for the types that accept one
+ /// (`varchar(10)`), except for `float(p)` where it selects the type. This is the one place
+ /// where accepting more than Postgres is deliberate: it also takes a modifier on a type
+ /// that cannot carry one (`int4(5)`, which Postgres rejects) rather than fail a cast a BI
+ /// tool would not have emitted in the first place.
+ fn regtype_name_to_oid(name: &str) -> Option {
+ /// Names accepted by the Postgres type name parser that are neither a `typname`
+ /// nor a canonical regtype spelling.
+ const ALIASES: &[(&str, &str)] = &[
+ ("int", "int4"),
+ ("decimal", "numeric"),
+ ("char", "bpchar"),
+ ("float", "float8"),
+ ];
+
+ let name = name.trim().to_lowercase();
+ let name = name.strip_prefix("pg_catalog.").unwrap_or(&name);
+ // A type modifier does not affect the resolved type: `varchar(10)::regtype` is `varchar`
+ let (name, typmod) = match (name.find('('), name.rfind(')')) {
+ (Some(open), Some(close)) if open < close => (
+ format!("{}{}", &name[..open], &name[close + 1..]),
+ Some(name[open + 1..close].trim().to_string()),
+ ),
+ _ => (name.to_string(), None),
+ };
+ let name = name.trim();
+ if name.is_empty() {
+ return None;
+ }
+
+ // `float(p)` is the one modifier that picks the type rather than the precision within
+ // it: `real` up to 24 bits of mantissa, `double precision` above, and Postgres rejects
+ // anything wider
+ if let (Some(typmod), "float") = (&typmod, name.trim_end_matches("[]")) {
+ let element = match typmod.parse::().ok()? {
+ 1..=24 => "float4",
+ 25..=53 => "float8",
+ _ => return None,
+ };
+
+ return Self::regtype_name_to_oid(&name.replace("float", element));
+ }
+
+ let canonical = ALIASES
+ .iter()
+ .find_map(|(alias, typname)| (*alias == name).then_some(*typname))
+ .unwrap_or(name);
+
+ if let Some(typ) = PgType::get_all()
+ .iter()
+ .find(|typ| typ.typname == canonical || typ.regtype == canonical)
+ {
+ return Some(typ.oid);
+ }
+
+ // `int4[]` is spelled `_int4` as a typname; the canonical regtype spelling
+ // (`integer[]`) is matched above. Postgres does not model the number of dimensions
+ // in the type either, so `int4[][]` resolves to the same array type
+ let elem = canonical.trim_end_matches("[]");
+ if elem == canonical {
+ return None;
+ }
+
+ let elem_oid = Self::regtype_name_to_oid(elem)?;
+ let elem_typ = PgType::get_all().iter().find(|typ| typ.oid == elem_oid)?;
+
+ (elem_typ.typarray != 0).then_some(elem_typ.typarray)
+ }
+
+ /// Splits a one dimensional Postgres array literal (`{a,b,"c,d"}`) into its elements.
+ /// `None` for anything this does not model: multi dimensional literals, an explicit
+ /// dimension prefix or an unterminated quote. Callers are expected to leave the cast
+ /// alone in that case rather than to substitute a partial array.
+ ///
+ /// One deviation from Postgres: whitespace surrounding a quoted element is kept
+ /// (`{ "int4" }` yields `" int4 "`), which is harmless as long as callers treat the
+ /// elements as type names, since name resolution trims.
+ fn parse_array_literal(input: &str) -> Option>> {
+ let input = input.trim();
+ let input = input.strip_prefix('{')?.strip_suffix('}')?;
+ // An empty literal is a usable result, not a failure: `x = ANY (ARRAY[])` plans and is
+ // false, which is what Postgres answers for an empty array. Covered end to end by the
+ // `empty_regtype_array_comparison` test
+ if input.trim().is_empty() {
+ return Some(vec![]);
+ }
+
+ let mut elements = vec![];
+ let mut current = String::new();
+ let mut quoted = false;
+ // Quoting and escaping both make an element a string, so that `NULL` and `"NULL"`
+ // (or `\N\U\L\L`) stay distinguishable
+ let mut escaped = false;
+ let mut chars = input.chars();
+
+ while let Some(c) = chars.next() {
+ match c {
+ '\\' => {
+ current.push(chars.next()?);
+ escaped = true;
+ }
+ '"' => {
+ quoted = !quoted;
+ escaped = true;
+ }
+ '{' | '}' if !quoted => return None,
+ ',' if !quoted => {
+ elements.push(Self::finish_array_element(current, escaped));
+ current = String::new();
+ escaped = false;
+ }
+ c => current.push(c),
+ }
+ }
+
+ if quoted {
+ return None;
+ }
+
+ elements.push(Self::finish_array_element(current, escaped));
+
+ Some(elements)
+ }
+
+ /// An unquoted `NULL` inside an array literal is the null element; a quoted or escaped
+ /// one is the string `NULL`.
+ fn finish_array_element(element: String, escaped: bool) -> Option {
+ if escaped {
+ return Some(element);
+ }
+
+ let element = element.trim();
+ if element.eq_ignore_ascii_case("null") {
+ return None;
+ }
+
+ Some(element.to_string())
+ }
+
+ /// Expands `'{int,int8}'::regtype[]` to `ARRAY[23, 20]`. BI tools use this shape to
+ /// test an OID column against a set of types, and neither the `regtype` element type
+ /// nor an array cast of a string literal is understood downstream.
+ fn replace_regtype_array_cast(cast_expr: &Expr) -> Option {
+ let Expr::Value(value) = cast_expr else {
+ return None;
+ };
+ let (Value::SingleQuotedString(str_val) | Value::DoubleQuotedString(str_val)) =
+ &value.value
+ else {
+ return None;
+ };
+
+ let mut elements = vec![];
+ for element in Self::parse_array_literal(str_val)? {
+ let element = match element {
+ None => Expr::Value(Value::Null.into()),
+ Some(name) => match Self::regtype_name_to_oid(&name) {
+ Some(oid) => Expr::Value(Value::Number(oid.to_string(), false).into()),
+ None => {
+ trace!(
+ r#"Unable to cast string to RegType[] via CastReplacer, type "{}" is not defined"#,
+ name
+ );
+
+ return None;
+ }
+ },
+ };
+
+ elements.push(element);
+ }
+
+ Some(Expr::Array(ast::Array {
+ elem: elements,
+ named: true,
+ }))
+ }
+
+ /// Operand of a `::regtype` or `::regtype[]` cast, ignoring parentheses.
+ fn regtype_cast_operand(expr: &Expr) -> Option<&Expr> {
+ match expr {
+ Expr::Nested(inner) => Self::regtype_cast_operand(inner),
+ Expr::Cast {
+ expr, data_type, ..
+ } => match data_type {
+ ast::DataType::Array(ast::ArrayElemTypeDef::SquareBracket(elem_type, _)) => {
+ Self::is_regtype(elem_type).then_some(&**expr)
+ }
+ data_type => Self::is_regtype(data_type).then_some(&**expr),
+ },
+ _ => None,
+ }
+ }
+
+ /// True for a `::regtype` or `::regtype[]` cast of a literal that resolves to OIDs, and
+ /// for an array of such casts (`ARRAY['int4'::regtype]`). Parentheses are ignored.
+ /// A literal that does not resolve is not one, so that a comparison against it is left
+ /// untouched as a whole instead of half rewritten.
+ fn is_resolvable_regtype_literal(expr: &Expr) -> bool {
+ match expr {
+ Expr::Nested(inner) => Self::is_resolvable_regtype_literal(inner),
+ // An empty `ARRAY[]` carries no regtype to recognize, unlike the empty cast form
+ // `'{}'::regtype[]`. Either way the comparison is false, with or without the cast
+ // on the column side, as `empty_regtype_array_comparison` shows
+ Expr::Array(array) => {
+ !array.elem.is_empty() && array.elem.iter().all(Self::is_resolvable_regtype_literal)
+ }
+ Expr::Cast {
+ expr, data_type, ..
+ } => match data_type {
+ ast::DataType::Array(ast::ArrayElemTypeDef::SquareBracket(elem_type, _))
+ if Self::is_regtype(elem_type) =>
+ {
+ Self::replace_regtype_array_cast(expr).is_some()
+ }
+ data_type if Self::is_regtype(data_type) => match &**expr {
+ Expr::Value(value) => match &value.value {
+ Value::SingleQuotedString(str_val) | Value::DoubleQuotedString(str_val) => {
+ Self::regtype_name_to_oid(str_val).is_some()
+ }
+ _ => false,
+ },
+ _ => false,
+ },
+ _ => false,
+ },
+ _ => false,
+ }
+ }
+
+ /// Drops the `::regtype` cast of a column compared against a regtype literal, as in
+ /// `a.atttypid::regtype = 'int4'::regtype`. Comparing two regtypes compares OIDs in
+ /// Postgres, and the literal side is rewritten to an OID; keeping the cast on the
+ /// column would instead compare the type name `format_type` renders with that OID,
+ /// which silently matches nothing.
+ ///
+ /// Only a cast on both sides is stripped: `a.atttypid::regtype = 'integer'` compares
+ /// against a plain string, so there the type name is the wanted shape.
+ ///
+ /// Callers cover the comparison contexts a BI tool is known to emit: a comparison
+ /// operator, `ANY`/`ALL`, `IN`, `BETWEEN` and a `CASE` operand. Anything else (say
+ /// `IS NOT DISTINCT FROM`) keeps the type name on the column side and so fails the same
+ /// way it did before regtype literals resolved at all.
+ fn strip_regtype_cast_for_oid_comparison(column_side: &mut Expr, literal_side: &Expr) {
+ if !Self::is_resolvable_regtype_literal(literal_side) {
+ return;
+ }
+
+ let column = Self::regtype_cast_operand(column_side)
+ .filter(|operand| matches!(operand, Expr::Identifier(_) | Expr::CompoundIdentifier(_)));
+ let Some(column) = column.cloned() else {
+ return;
+ };
+
+ *column_side = column;
+ }
}
impl<'ast> Visitor<'ast, ConnectionError> for CastReplacer {
+ fn transform_expr(&mut self, expr: &mut Expr) -> Result<(), ConnectionError> {
+ // Runs before the casts below are rewritten, so both sides are still visible as casts
+ match expr {
+ // Only the operators a regtype orders OIDs by, which is what makes stripping the
+ // cast on the column side equivalent to what Postgres compares
+ Expr::BinaryOp {
+ left,
+ op:
+ ast::BinaryOperator::Eq
+ | ast::BinaryOperator::NotEq
+ | ast::BinaryOperator::Lt
+ | ast::BinaryOperator::LtEq
+ | ast::BinaryOperator::Gt
+ | ast::BinaryOperator::GtEq,
+ right,
+ } => {
+ Self::strip_regtype_cast_for_oid_comparison(left, right);
+ Self::strip_regtype_cast_for_oid_comparison(right, left);
+ }
+ Expr::AnyOp {
+ left,
+ compare_op:
+ ast::BinaryOperator::Eq
+ | ast::BinaryOperator::NotEq
+ | ast::BinaryOperator::Lt
+ | ast::BinaryOperator::LtEq
+ | ast::BinaryOperator::Gt
+ | ast::BinaryOperator::GtEq,
+ right,
+ ..
+ }
+ | Expr::AllOp {
+ left,
+ compare_op:
+ ast::BinaryOperator::Eq
+ | ast::BinaryOperator::NotEq
+ | ast::BinaryOperator::Lt
+ | ast::BinaryOperator::LtEq
+ | ast::BinaryOperator::Gt
+ | ast::BinaryOperator::GtEq,
+ right,
+ } => {
+ Self::strip_regtype_cast_for_oid_comparison(left, right);
+ }
+ Expr::Between {
+ expr, low, high, ..
+ } => {
+ if Self::is_resolvable_regtype_literal(high) {
+ Self::strip_regtype_cast_for_oid_comparison(expr, low);
+ }
+ }
+ Expr::InList { expr, list, .. } => {
+ if let Some(first) = list.first() {
+ if list.iter().all(Self::is_resolvable_regtype_literal) {
+ Self::strip_regtype_cast_for_oid_comparison(expr, first);
+ }
+ }
+ }
+ // `CASE a.atttypid::regtype WHEN 'int4'::regtype THEN ... END` compares the operand
+ // against every condition
+ Expr::Case {
+ operand: Some(operand),
+ conditions,
+ ..
+ } => {
+ if let Some(first) = conditions.first() {
+ if conditions
+ .iter()
+ .all(|when| Self::is_resolvable_regtype_literal(&when.condition))
+ {
+ Self::strip_regtype_cast_for_oid_comparison(operand, &first.condition);
+ }
+ }
+ }
+ _ => (),
+ }
+
+ Ok(())
+ }
+
fn visit_cast(&mut self, expr: &mut Expr) -> Result<(), ConnectionError> {
if let Expr::Cast {
expr: cast_expr,
@@ -865,6 +1218,47 @@ impl<'ast> Visitor<'ast, ConnectionError> for CastReplacer {
} = expr
{
match data_type {
+ // Matched ahead of the `Custom` arm below so that the qualified spelling
+ // (`pg_catalog.regtype`) takes the same path as the bare one
+ _ if Self::is_regtype(data_type) => {
+ self.visit_expr(&mut *cast_expr)?;
+
+ match &**cast_expr {
+ // A regtype renders as the type's name. Comparisons against a regtype
+ // literal are rewritten to compare OIDs instead, see
+ // `strip_regtype_cast_for_oid_comparison`
+ Expr::Identifier(_) | Expr::CompoundIdentifier(_) => {
+ *expr = Expr::Function(new_function(
+ "format_type",
+ vec![
+ FunctionArg::Unnamed(FunctionArgExpr::Expr(*cast_expr.clone())),
+ FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Value(
+ Value::Null.into(),
+ ))),
+ ],
+ ))
+ }
+ // A regtype from a type name is that type's OID, which is what
+ // introspection queries compare against `pg_attribute.atttypid`
+ Expr::Value(val) => {
+ let Some(str_val) = self.parse_value_to_str(&val.value) else {
+ return Ok(());
+ };
+
+ match Self::regtype_name_to_oid(str_val) {
+ Some(oid) => {
+ *expr =
+ Expr::Value(Value::Number(oid.to_string(), false).into());
+ }
+ None => trace!(
+ r#"Unable to cast string to RegType via CastReplacer, type "{}" is not defined"#,
+ str_val
+ ),
+ }
+ }
+ _ => (),
+ }
+ }
ast::DataType::Custom(name, _) => match name.to_string().to_lowercase().as_str() {
"name" | "oid" | "information_schema.cardinal_number" | "regproc" => {
self.visit_expr(&mut *cast_expr)?;
@@ -906,21 +1300,6 @@ impl<'ast> Visitor<'ast, ConnectionError> for CastReplacer {
*data_type = ast::DataType::Timestamp(None, ast::TimezoneInfo::None);
}
- "regtype" => {
- self.visit_expr(&mut *cast_expr)?;
-
- if let Expr::Identifier(_) = &**cast_expr {
- *expr = Expr::Function(new_function(
- "format_type",
- vec![
- FunctionArg::Unnamed(FunctionArgExpr::Expr(*cast_expr.clone())),
- FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Value(
- Value::Null.into(),
- ))),
- ],
- ))
- }
- }
"\"char\"" => {
self.visit_expr(&mut *cast_expr)?;
@@ -989,6 +1368,15 @@ impl<'ast> Visitor<'ast, ConnectionError> for CastReplacer {
*data_type = ast::DataType::Varchar(len);
}
+ ast::DataType::Array(ast::ArrayElemTypeDef::SquareBracket(elem_type, _))
+ if Self::is_regtype(elem_type) =>
+ {
+ self.visit_expr(&mut *cast_expr)?;
+
+ if let Some(array) = Self::replace_regtype_array_cast(cast_expr) {
+ *expr = array;
+ }
+ }
ast::DataType::Regclass => match &**cast_expr {
Expr::Value(val) => {
let str_val = self.parse_value_to_str(&val.value);
@@ -1403,6 +1791,128 @@ mod tests {
Ok(())
}
+ #[test]
+ fn test_cast_replacer_regtype() -> Result<(), CubeError> {
+ // a type name cast to regtype resolves to that type's OID
+ run_cast_replacer("SELECT 'int4'::regtype", "SELECT 23")?;
+ run_cast_replacer("SELECT 'integer'::regtype", "SELECT 23")?;
+ run_cast_replacer("SELECT 'int'::regtype", "SELECT 23")?;
+ run_cast_replacer("SELECT 'pg_catalog.int8'::regtype", "SELECT 20")?;
+ run_cast_replacer("SELECT 'int8'::pg_catalog.regtype", "SELECT 20")?;
+ run_cast_replacer("SELECT 'character varying'::regtype", "SELECT 1043")?;
+ run_cast_replacer("SELECT 'varchar(10)'::regtype", "SELECT 1043")?;
+ // `float(p)` picks the type: real up to 24 bits of mantissa, double precision above
+ run_cast_replacer("SELECT 'float'::regtype", "SELECT 701")?;
+ run_cast_replacer("SELECT 'float(24)'::regtype", "SELECT 700")?;
+ run_cast_replacer("SELECT 'float(53)'::regtype", "SELECT 701")?;
+ run_cast_replacer("SELECT 'float(24)[]'::regtype", "SELECT 1021")?;
+ // Postgres rejects a wider mantissa, so no type resolves
+ run_cast_replacer("SELECT 'float(54)'::regtype", "SELECT 'float(54)'::regtype")?;
+ run_cast_replacer("SELECT 'int4[]'::regtype", "SELECT 1007")?;
+ run_cast_replacer("SELECT 'integer[]'::regtype", "SELECT 1007")?;
+ // Postgres does not track the number of dimensions in an array type
+ run_cast_replacer("SELECT 'int4[][]'::regtype", "SELECT 1007")?;
+ // an identifier keeps the display semantics of regtype
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype",
+ "SELECT format_type(a.atttypid, NULL)",
+ )?;
+ // an unknown type name is left alone
+ run_cast_replacer(
+ "SELECT 'unknown_type'::regtype",
+ "SELECT 'unknown_type'::regtype",
+ )?;
+
+ // array literals expand to an ARRAY of OIDs
+ run_cast_replacer(
+ "SELECT '{int,int8,int2}'::regtype[]",
+ "SELECT ARRAY[23, 20, 21]",
+ )?;
+ run_cast_replacer(
+ "SELECT '{ int4 , \"character varying\" }'::regtype[]",
+ "SELECT ARRAY[23, 1043]",
+ )?;
+ run_cast_replacer("SELECT '{int4,NULL}'::regtype[]", "SELECT ARRAY[23, NULL]")?;
+ // an empty array is a usable result: the comparison is false, as in Postgres
+ run_cast_replacer("SELECT '{}'::regtype[]", "SELECT ARRAY[]")?;
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype = ANY ('{}'::regtype[])",
+ "SELECT a.atttypid = ANY(ARRAY[])",
+ )?;
+ run_cast_replacer("SELECT '{int4}'::pg_catalog.regtype[]", "SELECT ARRAY[23]")?;
+ // an unknown element leaves the whole cast alone
+ run_cast_replacer(
+ "SELECT '{int4,unknown_type}'::regtype[]",
+ "SELECT '{int4,unknown_type}'::regtype[]",
+ )?;
+ // an escaped element is the string `NULL`, not the NULL element, and no type resolves
+ // from it, so the cast is left alone
+ run_cast_replacer(
+ r#"SELECT '{\N\U\L\L}'::regtype[]"#,
+ r#"SELECT '{\N\U\L\L}'::regtype[]"#,
+ )?;
+ // a literal this does not model leaves the cast alone rather than dropping elements
+ run_cast_replacer(
+ "SELECT '{{int4},{int8}}'::regtype[]",
+ "SELECT '{{int4},{int8}}'::regtype[]",
+ )?;
+ run_cast_replacer(
+ r#"SELECT '{"int4}'::regtype[]"#,
+ r#"SELECT '{"int4}'::regtype[]"#,
+ )?;
+
+ // comparing two regtypes compares OIDs, so the column keeps its OID here
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype = 'int8'::regtype",
+ "SELECT a.atttypid = 20",
+ )?;
+ run_cast_replacer(
+ "SELECT 'int8'::regtype = a.atttypid::regtype",
+ "SELECT 20 = a.atttypid",
+ )?;
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype = ANY ('{int8,int2}'::regtype[])",
+ "SELECT a.atttypid = ANY(ARRAY[20, 21])",
+ )?;
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype IN ('int8'::regtype, 'int2'::regtype)",
+ "SELECT a.atttypid IN (20, 21)",
+ )?;
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype = ANY (ARRAY['int8'::regtype, 'int2'::regtype])",
+ "SELECT a.atttypid = ANY(ARRAY[20, 21])",
+ )?;
+ // an operator a regtype does not order OIDs by keeps the type name on the column side
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype ~ ANY ('{int8}'::regtype[])",
+ "SELECT format_type(a.atttypid, NULL) ~ ANY(ARRAY[20])",
+ )?;
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype BETWEEN 'int2'::regtype AND 'int8'::regtype",
+ "SELECT a.atttypid BETWEEN 21 AND 20",
+ )?;
+ run_cast_replacer(
+ "SELECT CASE a.atttypid::regtype WHEN 'int8'::regtype THEN 1 END",
+ "SELECT CASE a.atttypid WHEN 20 THEN 1 END",
+ )?;
+ // a comparison against a plain string wants the type name instead
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype = 'bigint'",
+ "SELECT format_type(a.atttypid, NULL) = 'bigint'",
+ )?;
+ // a literal that resolves to no type leaves both sides of the comparison alone
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype = 'unknown_type'::regtype",
+ "SELECT format_type(a.atttypid, NULL) = 'unknown_type'::regtype",
+ )?;
+ run_cast_replacer(
+ "SELECT a.atttypid::regtype = ANY ('{int8,unknown_type}'::regtype[])",
+ "SELECT format_type(a.atttypid, NULL) = ANY('{int8,unknown_type}'::regtype[])",
+ )?;
+
+ Ok(())
+ }
+
fn run_redshift_date_part_replacer(input: &str, output: &str) -> Result<(), CubeError> {
let stmts = Parser::parse_sql(&PostgreSqlDialect {}, &input).unwrap();