From f7ff67aa508d0b6e6512d503ece7f7516567b906 Mon Sep 17 00:00:00 2001 From: waralexrom <108349432+waralexrom@users.noreply.github.com> Date: Wed, 5 Aug 2026 14:51:48 +0200 Subject: [PATCH] fix(tesseract): plan multiplied measures per cube and join tree (#11436) * fix(tesseract): plan multiplied measures per join tree, not per cube Multiplied measures were bucketed by owning cube and the whole bucket was then required to resolve to a single multi-fact join group, so two measures of one cube needing different join trees were refused with "Expected just one multi-fact join group". The key cube fixes the primary keys to deduplicate on and the join tree fixes the joins to build; neither determines the other. Group by both and emit one AggregateMultipliedSubquery per pair, mirroring the regular-measure branch, which already emits one CTE per join group. Trees are kept apart rather than merged so that a measure's value does not depend on which other measures share the query: merging sibling one-to-many branches introduces a cross product that neither original tree had. With a single group per cube the plan is unchanged, so queries that planned before are unaffected. * fix(tesseract): decompose calculated measures that reach other cubes Reaching a cube beyond its own makes a multiplied measure go through the measure-join subquery, which renders measures without their aggregate so the select above can re-aggregate them. A calculated measure has no aggregate of its own, so nothing was re-applied: its components lost theirs inside the subquery and the expression came out neither aggregated nor grouped. Mark such a measure composite instead, so its components travel in its place - each on the join tree its own definition asks for - and the expression is rebuilt above them. Only when the reach runs entirely through those components. A dimension dependency, a raw cube reference or no measure dependency at all leaves the measure with something to evaluate where it stands, and the select above has no cube joined in to evaluate it against; those keep going whole. * fix(tesseract): decompose calculated measures only where multiplied Decomposition exists to work around the measure-join subquery, which is built only for a measure that is multiplied by its join tree. Elsewhere the measure is read off a leaf-measure query that keeps its aggregate, and splitting it changes the answer: components can root at different cubes, so each divides by the rows its own leg sees rather than by the query's. Fail loudly instead of asserting when join grouping drops a measure of a bucket - a debug assertion leaves release builds emitting a plan with the measure missing. * test(tesseract): pin the same-cube split without a database The join grouping that multiplied measures rely on was only covered through Postgres-backed snapshots, so a run without the integration feature asserted nothing about it. Cover it where the grouping is decided. Record the denominator the non-multiplied control divides by: the ratio and the sibling total on that row are taken over different rows and are not meant to reconcile, which is not evident from the snapshot alone. * test(tesseract): assert the plan shape, not only the rows it returns The suite proved the numbers through Postgres but said nothing about how the plan reached them, so a run without the integration feature let a regression in the join grouping through as long as the query still planned. Count the deduplicating key subselects - one per (key cube, join tree) - and check whether a calculated measure's components get columns of their own. Both read structure out of the generated SQL rather than pinning it. * test(tesseract): state which planning strategy each case is holding A result snapshot records the numbers a query returned but not how it got them, so what these cases exist to pin was only in their names. Assert the leg count everywhere the strategy is the point: aggregated in place or deduplicated through the keys path, measures of one cube split across trees or grouped onto one. The mixed-shapes case wanted three legs and gets two - the calculated measure shares a tree with the bare aggregate, which is grouping working rather than a shape being missed. * test(tesseract): anchor the negated assertions and scope the grouping unit test A negated substring match passes both when a measure genuinely stays whole and when the alias convention the helper reads stops holding, at which point the assertion is inert without failing. Pair each with the positive case from the same query, which cannot hold unless the convention does. Say what the same-cube grouping unit test covers - the precondition callers slice on, not what any of them emits per group - and correct the sibling fan-out case, whose prose claimed a deduplication its own assertion denies. * fix(tesseract): refuse measures the measure subquery cannot carry Splitting a calculated measure into its components put the two halves of an expression on separate legs. When those legs root at different cubes they see different rows, so the ratio came out over a denominator its numerator never saw - answered, with no error, differently depending on whether the query carried a fan-out dimension. The split also never covered the shape it was written for: whether the measure subquery is built is decided per join group, while the condition for splitting was read off a single measure, so a calculated measure whose own dependencies stay inside its cube still lands in that subquery whenever a group member reaches out of it. Refuse instead. A measure carrying no aggregate of its own cannot survive a subquery that renders measures without theirs, whichever way it got there, so say that where the subquery is built and name the measure. Shapes that used to reach the database as SQL it rejects now stop at the planner. * fix(tesseract): check the measure subquery where it is rendered Refusing during logical planning ran ahead of the pre-aggregation optimizer, which may replace the whole node with a rollup and never render the subquery at all - a query a rollup answers was turned into an error. Move the check to where the subquery is actually built, past the point the override returns. Say what the user can do about it. The subquery is forced by whichever measure of the group reaches another cube, which need not be the one being refused, so telling that measure to be rewritten over its own cube asks for something it may already satisfy; ask for a separate query instead. * test(tesseract): cover the rollup that renders no measure subquery Nothing held the check to where it runs: the fixture declared no rollups, so moving it back ahead of the pre-aggregation optimizer left the suite green while turning a query a rollup answers into an error. Store one of the refused shapes and assert the usage. Name the reach rather than a sibling in the message. Four of the five refused shapes are single-measure queries where the measure reaches out itself, so asking for it in a separate query sends the user round in a circle; what forces the subquery is the reach, whoever carries it. Say that member expressions are left out deliberately, so the skip does not read as an oversight. --- .../calculated-measure-multi-fact.test.ts | 211 +++++++ .../aggregate_multiplied_subquery.rs | 27 + .../src/planner/multi_fact_join_groups.rs | 43 ++ .../multiplied_measures_query_planner.rs | 81 ++- .../integration_calculated_multi_fact.yaml | 194 ++++++ ...tegration_calculated_multi_fact_tables.sql | 86 +++ .../integration/calculated_multi_fact.rs | 585 ++++++++++++++++++ .../src/tests/integration/mod.rs | 1 + ...ed_measure_alone_by_fan_out_dimension.snap | 9 + ...asure_components_by_fan_out_dimension.snap | 9 + ...measure_over_sums_with_joined_measure.snap | 9 + ...ng_other_cubes_without_multiplication.snap | 8 + ...measure_reading_its_own_cube_directly.snap | 8 + ...alculated_measure_with_joined_measure.snap | 9 + ...e_with_joined_measure_without_fan_out.snap | 8 + ...h_distinct_count_by_fan_out_dimension.snap | 9 + ...measures_of_two_sibling_fan_out_cubes.snap | 8 + ...several_join_tree_shapes_in_one_query.snap | 9 + ...plit_groups_rooted_at_different_cubes.snap | 9 + ...__three_nested_join_trees_of_one_cube.snap | 9 + ...sibling_lookup_join_trees_of_one_cube.snap | 9 + ...s_of_one_cube_needing_different_joins.snap | 9 + 22 files changed, 1320 insertions(+), 30 deletions(-) create mode 100644 packages/cubejs-schema-compiler/test/integration/postgres/calculated-measure-multi-fact.test.ts create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/test_fixtures/schemas/yaml_files/common/integration_calculated_multi_fact.yaml create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/test_fixtures/schemas/yaml_files/seeds/integration_calculated_multi_fact_tables.sql create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_alone_by_fan_out_dimension.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_components_by_fan_out_dimension.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_over_sums_with_joined_measure.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_reaching_other_cubes_without_multiplication.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_reading_its_own_cube_directly.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_with_joined_measure.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_with_joined_measure_without_fan_out.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__joined_measure_with_distinct_count_by_fan_out_dimension.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__measures_of_two_sibling_fan_out_cubes.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__several_join_tree_shapes_in_one_query.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__split_groups_rooted_at_different_cubes.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__three_nested_join_trees_of_one_cube.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__two_sibling_lookup_join_trees_of_one_cube.snap create mode 100644 rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__two_sums_of_one_cube_needing_different_joins.snap diff --git a/packages/cubejs-schema-compiler/test/integration/postgres/calculated-measure-multi-fact.test.ts b/packages/cubejs-schema-compiler/test/integration/postgres/calculated-measure-multi-fact.test.ts new file mode 100644 index 0000000000000..8e5aab7136a9c --- /dev/null +++ b/packages/cubejs-schema-compiler/test/integration/postgres/calculated-measure-multi-fact.test.ts @@ -0,0 +1,211 @@ +import { getEnv } from '@cubejs-backend/shared'; +import { PostgresQuery } from '../../../src/adapter/PostgresQuery'; +import { prepareJsCompiler } from '../../unit/PrepareCompiler'; +import { dbRunner } from './PostgresDBRunner'; + +// Calculated measures (`type: number` over other measures of the same cube) +// combined with a dimension reached through a hasMany join. The fan-out forces +// every measure to pick a strategy: countDistinct survives row multiplication +// and is aggregated in place, sum has to go through the keys subquery. A +// calculated measure is neither - it is an expression over aggregates and can +// only be evaluated once its components have been re-aggregated. +describe('Calculated measure on the multi-fact path', () => { + jest.setTimeout(200000); + + const { compiler, joinGraph, cubeEvaluator } = prepareJsCompiler(` +cube(\`Payments\`, { + // id is TEXT on purpose: it is the operand that ends up in arithmetic when a + // calculated measure loses the aggregation around its components. + sql: \` + SELECT 'p1' AS id, 'SUCCESS' AS status, 100 AS amount, 'EUR' AS currency UNION ALL + SELECT 'p2' AS id, 'SUCCESS' AS status, 200 AS amount, 'EUR' AS currency UNION ALL + SELECT 'p3' AS id, 'DECLINED' AS status, 300 AS amount, 'EUR' AS currency UNION ALL + SELECT 'p4' AS id, 'SUCCESS' AS status, 400 AS amount, 'USD' AS currency + \`, + + joins: { + Meta: { + relationship: \`hasMany\`, + sql: \`\${CUBE}.id = \${Meta}.payment_id\`, + }, + Rates: { + relationship: \`belongsTo\`, + sql: \`\${CUBE}.currency = \${Rates}.currency\`, + }, + }, + + measures: { + count: { + sql: \`id\`, + type: \`countDistinct\`, + }, + successCount: { + sql: \`id\`, + type: \`countDistinct\`, + filters: [{ sql: \`\${CUBE}.status = 'SUCCESS'\` }], + }, + totalAmount: { + sql: \`amount\`, + type: \`sum\`, + }, + successAmount: { + sql: \`amount\`, + type: \`sum\`, + filters: [{ sql: \`\${CUBE}.status = 'SUCCESS'\` }], + }, + // Needs a join to Rates, and sum is not immune to the Meta fan-out, so it + // takes the keys-subquery path. + convertedValue: { + sql: \`\${CUBE}.amount / nullif(\${Rates.fxRate}, 0)\`, + type: \`sum\`, + }, + // Calculated measures over components of the same cube. The components + // differ in whether they survive row multiplication on their own: + // countDistinct does, sum does not. + successRate: { + sql: \`100.0 * \${successCount} / nullif(\${count}, 0)\`, + type: \`number\`, + }, + successAmountRate: { + sql: \`100.0 * \${successAmount} / nullif(\${totalAmount}, 0)\`, + type: \`number\`, + }, + }, + + dimensions: { + id: { sql: \`id\`, type: \`string\`, primaryKey: true }, + status: { sql: \`status\`, type: \`string\` }, + }, +}); + +cube(\`Meta\`, { + // p1 carries two meta rows so grouping by Meta.value multiplies it. + sql: \` + SELECT 'm1' AS id, 'p1' AS payment_id, 'A' AS value UNION ALL + SELECT 'm1b' AS id, 'p1' AS payment_id, 'A' AS value UNION ALL + SELECT 'm2' AS id, 'p2' AS payment_id, 'A' AS value UNION ALL + SELECT 'm3' AS id, 'p3' AS payment_id, 'A' AS value UNION ALL + SELECT 'm4' AS id, 'p4' AS payment_id, 'B' AS value + \`, + dimensions: { + id: { sql: \`id\`, type: \`string\`, primaryKey: true }, + paymentId: { sql: \`payment_id\`, type: \`string\` }, + value: { sql: \`value\`, type: \`string\` }, + }, +}); + +cube(\`Rates\`, { + sql: \` + SELECT 'EUR' AS currency, 1.0 AS fx_rate UNION ALL + SELECT 'USD' AS currency, 2.0 AS fx_rate + \`, + dimensions: { + currency: { sql: \`currency\`, type: \`string\`, primaryKey: true }, + fxRate: { sql: \`fx_rate\`, type: \`number\` }, + }, +}); + `); + + async function runQuery(q) { + await compiler.compile(); + const query = new PostgresQuery({ joinGraph, cubeEvaluator, compiler }, q); + return dbRunner.testQuery(query.buildSqlAndParams()); + } + + async function expectQueryToFail(q) { + await compiler.compile(); + try { + const query = new PostgresQuery({ joinGraph, cubeEvaluator, compiler }, q); + await dbRunner.testQuery(query.buildSqlAndParams()); + } catch (e: any) { + return e.message as string; + } + throw new Error('Expected the query to fail, but it succeeded'); + } + + it('calculated measure alone, grouped by a fan-out dimension', async () => { + expect(await runQuery({ + measures: ['Payments.successRate'], + dimensions: ['Meta.value'], + order: [{ id: 'Meta.value' }], + })).toEqual([ + { meta__value: 'A', payments__success_rate: '66.6666666666666667' }, + { meta__value: 'B', payments__success_rate: '100.0000000000000000' }, + ]); + }); + + it('calculated measure components, grouped by a fan-out dimension', async () => { + expect(await runQuery({ + measures: ['Payments.successCount', 'Payments.count'], + dimensions: ['Meta.value'], + order: [{ id: 'Meta.value' }], + })).toEqual([ + { meta__value: 'A', payments__success_count: '2', payments__count: '3' }, + { meta__value: 'B', payments__success_count: '1', payments__count: '1' }, + ]); + }); + + it('joined measure next to a distinct count, grouped by a fan-out dimension', async () => { + expect(await runQuery({ + measures: ['Payments.convertedValue', 'Payments.count'], + dimensions: ['Meta.value'], + order: [{ id: 'Meta.value' }], + })).toEqual([ + { meta__value: 'A', payments__converted_value: '600.0000000000000000', payments__count: '3' }, + { meta__value: 'B', payments__converted_value: '200.0000000000000000', payments__count: '1' }, + ]); + }); + + it('calculated measure next to a joined measure, grouped by a fan-out dimension', async () => { + const query = { + measures: ['Payments.successRate', 'Payments.convertedValue'], + dimensions: ['Meta.value'], + order: [{ id: 'Meta.value' }], + }; + + if (!getEnv('nativeSqlPlanner')) { + // The calculated measure is inlined into the ungrouped measure-join with + // the aggregation around its components removed, leaving the TEXT id + // column in arithmetic. + expect(await expectQueryToFail(query)).toContain('operator does not exist: numeric * text'); + return; + } + + expect(await runQuery(query)).toEqual([ + { meta__value: 'A', payments__success_rate: '66.6666666666666667', payments__converted_value: '600.0000000000000000' }, + { meta__value: 'B', payments__success_rate: '100.0000000000000000', payments__converted_value: '200.0000000000000000' }, + ]); + }); + + it('calculated measure over sums next to a joined measure, grouped by a fan-out dimension', async () => { + const query = { + measures: ['Payments.successAmountRate', 'Payments.convertedValue'], + dimensions: ['Meta.value'], + order: [{ id: 'Meta.value' }], + }; + + if (!getEnv('nativeSqlPlanner')) { + // Types line up here, so the failure surfaces one step later: the + // calculated measure is projected without an aggregate and without being + // grouped. + expect(await expectQueryToFail(query)).toContain('must appear in the GROUP BY clause'); + return; + } + + expect(await runQuery(query)).toEqual([ + { meta__value: 'A', payments__success_amount_rate: '50.0000000000000000', payments__converted_value: '600.0000000000000000' }, + { meta__value: 'B', payments__success_amount_rate: '100.0000000000000000', payments__converted_value: '200.0000000000000000' }, + ]); + }); + + it('calculated measure next to a joined measure, without a fan-out dimension', async () => { + expect(await runQuery({ + measures: ['Payments.successRate', 'Payments.convertedValue'], + dimensions: ['Payments.status'], + order: [{ id: 'Payments.status' }], + })).toEqual([ + { payments__status: 'DECLINED', payments__success_rate: '0.00000000000000000000', payments__converted_value: '300.0000000000000000' }, + { payments__status: 'SUCCESS', payments__success_rate: '100.0000000000000000', payments__converted_value: '500.0000000000000000' }, + ]); + }); +}); diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/physical_plan_builder/processors/aggregate_multiplied_subquery.rs b/rust/cube/cubesqlplanner/cubesqlplanner/src/physical_plan_builder/processors/aggregate_multiplied_subquery.rs index 96a198f867b38..00d42e07168d1 100644 --- a/rust/cube/cubesqlplanner/cubesqlplanner/src/physical_plan_builder/processors/aggregate_multiplied_subquery.rs +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/physical_plan_builder/processors/aggregate_multiplied_subquery.rs @@ -9,9 +9,35 @@ use crate::physical_plan::{ }; use crate::physical_plan_builder::PhysicalPlanBuilder; use crate::planner::MeasureRenderModifier; +use crate::planner::{AggregateWrap, MemberSymbol}; use cubenativeutils::CubeError; use std::rc::Rc; +/// The measure subquery renders measures without their aggregate for the select +/// above to re-apply. A measure carrying none of its own has nothing to +/// re-apply and would come out neither aggregated nor grouped. +/// +/// Member expressions are left out on purpose: the SQL API builds them ad-hoc +/// and their aggregation is not described by a measure kind. +fn check_measures_survive_measure_subquery(measures: &[Rc]) -> Result<(), CubeError> { + for measure in measures.iter() { + let Ok(symbol) = measure.as_measure() else { + continue; + }; + if matches!(symbol.kind().aggregate_wrap(), AggregateWrap::PassThrough) { + return Err(CubeError::user(format!( + "{} has no aggregate of its own, so it cannot be re-aggregated over the \ + deduplicated rows this query needs - a measure of its group reaches another \ + cube, under a dimension that multiplies its rows. Please drop the multiplying \ + dimension, request the measures that reach out separately, or move the \ + aggregation into a measure.", + measure.full_name() + ))); + } + } + Ok(()) +} + pub struct AggregateMultipliedSubqueryProcessor<'a> { builder: &'a PhysicalPlanBuilder, } @@ -120,6 +146,7 @@ impl<'a> LogicalNodeProcessor<'a, AggregateMultipliedSubquery> } } AggregateMultipliedSubquerySource::MeasureSubquery(measure_subquery) => { + check_measures_survive_measure_subquery(&measure_subquery.schema.measures)?; let subquery = self .builder .process_node(measure_subquery.as_ref(), context)?; diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/planner/multi_fact_join_groups.rs b/rust/cube/cubesqlplanner/cubesqlplanner/src/planner/multi_fact_join_groups.rs index 6231c9cb26457..425f6eca3743d 100644 --- a/rust/cube/cubesqlplanner/cubesqlplanner/src/planner/multi_fact_join_groups.rs +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/planner/multi_fact_join_groups.rs @@ -559,6 +559,49 @@ mod tests { assert!(groups.single_join().is_err()); } + /// Two measures of the same cube can still need different join trees, and + /// then they are two groups like any other multi-fact pair - the owning cube + /// says nothing about which joins to build. + /// + /// This is the precondition callers who slice by owning cube depend on, not + /// a claim about what they do with it: what each planner emits per group is + /// its own to cover. + #[test] + fn test_two_groups_for_measures_of_one_cube() { + let schema = MockSchema::from_yaml_file("common/integration_calculated_multi_fact.yaml"); + let ctx = TestContext::new(schema).unwrap(); + + let total_amount = ctx.create_symbol("payments.total_amount").unwrap(); + let converted_value = ctx.create_symbol("payments.converted_value").unwrap(); + let meta_value = ctx.create_symbol("payment_meta.value").unwrap(); + + assert_eq!(total_amount.cube_name(), converted_value.cube_name()); + + let hints = MeasuresJoinHints::builder(&JoinHints::new()) + .add_dimensions(&[meta_value]) + .build(&[total_amount.clone(), converted_value.clone()]) + .unwrap(); + + let groups = MultiFactJoinGroups::try_new(ctx.query_tools().clone(), hints).unwrap(); + + assert!(groups.is_multi_fact()); + assert_eq!(groups.num_groups(), 2); + assert!(groups.single_join().is_err()); + + let grouped = groups + .groups() + .iter() + .map(|(_, measures)| measures.iter().map(|m| m.full_name()).collect::>()) + .collect::>(); + assert_eq!( + grouped, + vec![ + vec!["payments.total_amount"], + vec!["payments.converted_value"] + ] + ); + } + #[test] fn test_resolve_join_path_for_measure() { let schema = MockSchema::from_yaml_file("common/multi_fact.yaml"); diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/planner/planners/multiplied_measures_query_planner.rs b/rust/cube/cubesqlplanner/cubesqlplanner/src/planner/planners/multiplied_measures_query_planner.rs index b20eb9bfdc0a9..79ec917f2c0cf 100644 --- a/rust/cube/cubesqlplanner/cubesqlplanner/src/planner/planners/multiplied_measures_query_planner.rs +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/planner/planners/multiplied_measures_query_planner.rs @@ -17,8 +17,9 @@ use std::rc::Rc; /// Plans the per-measure CTEs that feed `FullKeyAggregate` in /// non-simple queries: regular measures become `MultiStageLeafMeasure` /// CTEs (one per multi-fact group), and multiplied measures become -/// `AggregateMultipliedSubquery` CTEs (one per owning cube). Both -/// kinds are registered into the shared `PlanningScope`. +/// `AggregateMultipliedSubquery` CTEs (one per owning cube and +/// multi-fact group). Both kinds are registered into the shared +/// `PlanningScope`. pub struct MultipliedMeasuresQueryPlanner { query_tools: Rc, query_properties: Rc, @@ -48,9 +49,9 @@ impl MultipliedMeasuresQueryPlanner { /// Registers per-measure CTEs into `scope`: regular measures /// become leaf-measure CTEs grouped by multi-fact join, multiplied /// measures become `AggregateMultipliedSubquery` CTEs grouped by - /// owning cube. Returns the subquery refs the caller's - /// `FullKeyAggregate` joins over. Errors if called on a simple - /// query. + /// owning cube and then by multi-fact join. Returns the subquery + /// refs the caller's `FullKeyAggregate` joins over. Errors if + /// called on a simple query. pub fn plan_queries( &self, scope: &mut PlanningScope, @@ -114,30 +115,50 @@ impl MultipliedMeasuresQueryPlanner { let join_multi_fact_groups = self .query_properties .compute_join_multi_fact_groups_with_measures(&measures)?; - let join = join_multi_fact_groups.single_join()?.ok_or_else(|| { - CubeError::internal("No join groups returned for aggregate measures".to_string()) - })?; - let aggregate_subquery_logical_plan = - self.aggregate_subquery_plan(&cube_name, &measures, join, scope)?; - - let cte_name = scope.next_cte_name(); - let member = Rc::new(LogicalMultiStageMember { - name: cte_name.clone(), - member_type: MultiStageMemberLogicalType::MultipliedMeasure( - aggregate_subquery_logical_plan.clone(), - ), - }); - scope.add_member(member); - - let ref_schema = aggregate_subquery_logical_plan.schema.clone(); - let subquery_ref = Rc::new( - MultiStageSubqueryRef::builder() - .name(cte_name.clone()) - .symbols(measures.clone()) - .schema(ref_schema) - .build(), - ); - subquery_refs.push(subquery_ref); + if join_multi_fact_groups.groups().is_empty() { + return Err(CubeError::internal(format!( + "No join groups returned for aggregate measures of cube {cube_name}" + ))); + } + // Every measure of the bucket must land in some group, otherwise it + // would be missing from the subquery refs the caller joins over. + let grouped_count: usize = join_multi_fact_groups + .groups() + .iter() + .map(|(_, ms)| ms.len()) + .sum(); + if grouped_count != measures.len() { + return Err(CubeError::internal(format!( + "Join grouping dropped measures of cube {cube_name}: {grouped_count} grouped of {}", + measures.len() + ))); + } + // The key cube fixes the primary keys to deduplicate on, the join + // tree fixes the joins to build. Measures of one cube can still + // need different trees, so each tree gets its own subquery. + for (join, group_measures) in join_multi_fact_groups.groups().iter() { + let aggregate_subquery_logical_plan = + self.aggregate_subquery_plan(&cube_name, group_measures, join.clone(), scope)?; + + let cte_name = scope.next_cte_name(); + let member = Rc::new(LogicalMultiStageMember { + name: cte_name.clone(), + member_type: MultiStageMemberLogicalType::MultipliedMeasure( + aggregate_subquery_logical_plan.clone(), + ), + }); + scope.add_member(member); + + let ref_schema = aggregate_subquery_logical_plan.schema.clone(); + let subquery_ref = Rc::new( + MultiStageSubqueryRef::builder() + .name(cte_name.clone()) + .symbols(group_measures.clone()) + .schema(ref_schema) + .build(), + ); + subquery_refs.push(subquery_ref); + } } Ok(subquery_refs) @@ -228,7 +249,7 @@ impl MultipliedMeasuresQueryPlanner { .get(key_cube_name) .unwrap_or(&false) { - return Err(CubeError::user(format!("{}' references cubes ({}) that lead to row multiplication. Please rewrite it using sub query.", measure.full_name(), cubes.join(", ")))); + return Err(CubeError::user(format!("{} references cubes ({}) that lead to row multiplication. Please rewrite it using sub query.", measure.full_name(), cubes.join(", ")))); } return Ok(true); } diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/test_fixtures/schemas/yaml_files/common/integration_calculated_multi_fact.yaml b/rust/cube/cubesqlplanner/cubesqlplanner/src/test_fixtures/schemas/yaml_files/common/integration_calculated_multi_fact.yaml new file mode 100644 index 0000000000000..ba2edfacce7d5 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/test_fixtures/schemas/yaml_files/common/integration_calculated_multi_fact.yaml @@ -0,0 +1,194 @@ +cubes: + - name: rates + sql: "SELECT * FROM rates" + dimensions: + - name: currency + type: string + sql: currency + primary_key: true + - name: fx_rate + type: number + sql: fx_rate + + # Declares the join towards `payments`, so a tree that needs this cube + # roots here rather than at `payments`. + - name: customers + sql: "SELECT * FROM customers" + joins: + - name: payments + relationship: one_to_many + sql: "{customers}.id = {payments.customer_id}" + dimensions: + - name: id + type: string + sql: id + primary_key: true + - name: tier + type: string + sql: tier + + - name: merchants + sql: "SELECT * FROM merchants" + dimensions: + - name: id + type: string + sql: id + primary_key: true + - name: name + type: string + sql: name + - name: commission + type: number + sql: commission + + - name: payment_meta + sql: "SELECT * FROM payment_meta" + dimensions: + - name: id + type: string + sql: id + primary_key: true + - name: payment_id + type: string + sql: payment_id + - name: value + type: string + sql: value + measures: + - name: count + type: count + + # Second one-to-many branch off `payments`, so a query can pull two + # fan-out siblings into one join tree. + - name: payment_tags + sql: "SELECT * FROM payment_tags" + dimensions: + - name: id + type: string + sql: id + primary_key: true + - name: payment_id + type: string + sql: payment_id + - name: tag + type: string + sql: tag + measures: + - name: count + type: count + + - name: payments + sql: "SELECT * FROM payments" + # Stores a measure the planner cannot compute through the measure + # subquery, at the grain that would force one. + pre_aggregations: + - name: max_fx_by_meta_value + type: rollup + measures: + - max_fx_rate + dimensions: + - payment_meta.value + joins: + - name: payment_meta + relationship: one_to_many + sql: "{payments}.id = {payment_meta.payment_id}" + - name: payment_tags + relationship: one_to_many + sql: "{payments}.id = {payment_tags.payment_id}" + - name: rates + relationship: many_to_one + sql: "{payments}.currency = {rates.currency}" + - name: merchants + relationship: many_to_one + sql: "{payments}.merchant_id = {merchants.id}" + dimensions: + - name: id + type: string + sql: id + primary_key: true + - name: status + type: string + sql: status + - name: customer_id + type: string + sql: customer_id + measures: + - name: count + type: count_distinct + sql: id + - name: success_count + type: count_distinct + sql: id + filters: + - sql: "{CUBE}.status = 'SUCCESS'" + - name: total_amount + type: sum + sql: amount + - name: success_amount + type: sum + sql: amount + filters: + - sql: "{CUBE}.status = 'SUCCESS'" + # Needs a join to `rates`, and `sum` is not immune to the + # `payment_meta` fan-out, so it takes the keys-subquery path. + - name: converted_value + type: sum + sql: "{CUBE}.amount / NULLIF({rates.fx_rate}, 0)" + # Calculated measures over components of the same cube. The + # components differ in whether they survive row multiplication on + # their own: count_distinct does, sum does not. + - name: success_rate + type: number + sql: "100.0 * {success_count} / NULLIF({count}, 0)" + - name: success_amount_rate + type: number + sql: "100.0 * {success_amount} / NULLIF({total_amount}, 0)" + # Needs `merchants` - a different single-cube extension than + # `converted_value`, so the two land on different join trees. + - name: commissioned_value + type: sum + sql: "{CUBE}.amount * {merchants.commission}" + # Reads three cubes at once. + - name: net_value + type: sum + sql: "{CUBE}.amount / NULLIF({rates.fx_rate}, 0) * {merchants.commission}" + # Reaches the second fan-out branch. + - name: tagged_amount + type: sum + sql: amount + filters: + - sql: "{payment_tags.tag} = 'vip'" + # Calculated measure whose components need *different* extra cubes, + # so its own footprint is their union. + - name: rate_vs_commission + type: number + sql: "100.0 * {converted_value} / NULLIF({commissioned_value}, 0)" + # Every dependency is a measure, but the expression also reads the + # cube's own table directly - that read has nowhere to go if the + # component leaves. + - name: converted_per_max_amount + type: number + sql: "{converted_value} / NULLIF(MAX({CUBE}.amount), 0)" + # Reaches `rates` past its component measure, so the component cannot + # be computed on its own: the leftover expression would have nothing + # to read `fx_rate` from. + - name: amount_over_fx + type: number + sql: "{total_amount} / NULLIF(MAX({rates.fx_rate}), 0)" + # Reaches `rates` with no component measure at all. + - name: max_fx_rate + type: number + sql: "MAX({rates.fx_rate})" + # Reaches a cube that owns the join towards `payments`, so this + # measure's tree roots away from the key cube. + - name: gold_amount + type: sum + sql: amount + filters: + - sql: "{customers.tier} = 'gold'" + # Components root at different cubes - `gold_amount` at `customers`, + # `total_amount` at `payments` - so splitting them would divide by a + # different set of rows than evaluating the ratio in one place. + - name: gold_share + type: number + sql: "100.0 * {gold_amount} / NULLIF({total_amount}, 0)" diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/test_fixtures/schemas/yaml_files/seeds/integration_calculated_multi_fact_tables.sql b/rust/cube/cubesqlplanner/cubesqlplanner/src/test_fixtures/schemas/yaml_files/seeds/integration_calculated_multi_fact_tables.sql new file mode 100644 index 0000000000000..038c6a25574d7 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/test_fixtures/schemas/yaml_files/seeds/integration_calculated_multi_fact_tables.sql @@ -0,0 +1,86 @@ +DROP TABLE IF EXISTS payment_tags CASCADE; +DROP TABLE IF EXISTS payment_meta CASCADE; +DROP TABLE IF EXISTS payments CASCADE; +DROP TABLE IF EXISTS merchants CASCADE; +DROP TABLE IF EXISTS customers CASCADE; +DROP TABLE IF EXISTS rates CASCADE; + +CREATE TABLE customers ( + id TEXT PRIMARY KEY, + tier TEXT NOT NULL +); + +CREATE TABLE rates ( + currency TEXT PRIMARY KEY, + fx_rate NUMERIC(10, 4) NOT NULL +); + +CREATE TABLE merchants ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + commission NUMERIC(10, 4) NOT NULL +); + +-- `id` is TEXT on purpose: it is the operand that ends up in arithmetic +-- when a calculated measure loses the aggregation around its components. +CREATE TABLE payments ( + id TEXT PRIMARY KEY, + status TEXT NOT NULL, + amount NUMERIC(10, 2) NOT NULL, + currency TEXT NOT NULL REFERENCES rates(currency), + merchant_id TEXT NOT NULL REFERENCES merchants(id), + -- nullable: p3 has no customer, so a tree rooted at `customers` cannot + -- reach it + customer_id TEXT REFERENCES customers(id), + created_at TIMESTAMP NOT NULL +); + +CREATE TABLE payment_meta ( + id TEXT PRIMARY KEY, + payment_id TEXT NOT NULL REFERENCES payments(id), + value TEXT NOT NULL +); + +CREATE TABLE payment_tags ( + id TEXT PRIMARY KEY, + payment_id TEXT NOT NULL REFERENCES payments(id), + tag TEXT NOT NULL +); + +INSERT INTO rates (currency, fx_rate) VALUES + ('EUR', 1.0), + ('USD', 2.0); + +INSERT INTO merchants (id, name, commission) VALUES + ('mer1', 'Acme', 0.1), + ('mer2', 'Globex', 0.5); + +INSERT INTO customers (id, tier) VALUES + ('c1', 'gold'), + ('c2', 'silver'); + +INSERT INTO payments (id, status, amount, currency, merchant_id, customer_id, created_at) VALUES + ('p1', 'SUCCESS', 100.00, 'EUR', 'mer1', 'c1', '2025-01-01 00:00:00'), + ('p2', 'SUCCESS', 200.00, 'EUR', 'mer1', 'c1', '2025-01-02 00:00:00'), + ('p3', 'DECLINED', 300.00, 'EUR', 'mer2', NULL, '2025-01-03 00:00:00'), + ('p4', 'SUCCESS', 400.00, 'USD', 'mer2', 'c2', '2025-01-04 00:00:00'), + -- sole payment under meta value 'C', and it has no customer: a tree rooted + -- at `customers` cannot reach it, so that leg reports no 'C' at all + ('p5', 'SUCCESS', 500.00, 'EUR', 'mer1', NULL, '2025-01-05 00:00:00'); + +-- p1 carries two meta rows so grouping by `payment_meta.value` multiplies it. +INSERT INTO payment_meta (id, payment_id, value) VALUES + ('m1', 'p1', 'A'), + ('m1b', 'p1', 'A'), + ('m2', 'p2', 'A'), + ('m3', 'p3', 'A'), + ('m4', 'p4', 'B'), + ('m5', 'p5', 'C'); + +-- p1 also carries two tags, so pulling both branches into one tree squares +-- its rows. +INSERT INTO payment_tags (id, payment_id, tag) VALUES + ('t1', 'p1', 'vip'), + ('t2', 'p1', 'new'), + ('t3', 'p2', 'vip'), + ('t4', 'p4', 'new'); diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs new file mode 100644 index 0000000000000..3e3a222ebf99f --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs @@ -0,0 +1,585 @@ +//! Measures of one cube that need different join trees, combined with a +//! dimension reached through a one-to-many join. +//! +//! The fan-out makes every measure pick a strategy: `count_distinct` survives +//! row multiplication and is aggregated in place, while `sum` has to go through +//! the keys subquery that deduplicates it. The key cube says which primary keys +//! to deduplicate on and the join tree says which joins to build, so measures +//! sharing a cube can still need a subquery each. +//! +//! A calculated measure (`type: number`) carries no aggregate of its own. It +//! rides along wherever its own tree takes it, but it cannot survive the +//! measure subquery that reaching another cube forces - there is nothing to +//! re-apply above it - and those shapes are refused while planning. + +use crate::test_fixtures::cube_bridge::MockSchema; +use crate::test_fixtures::test_utils::TestContext; +use indoc::indoc; + +fn create_context() -> TestContext { + let schema = MockSchema::from_yaml_file("common/integration_calculated_multi_fact.yaml"); + TestContext::new(schema).unwrap() +} + +const SEED: &str = "integration_calculated_multi_fact_tables.sql"; + +// One per (key cube, join tree), each rendering its key set as a `keys` +// subselect. Counting them states which strategy a test holds the planner to, +// which the result snapshot alone cannot. +fn keys_subquery_count(sql: &str) -> usize { + sql.matches(r#" AS "keys""#).count() +} + +fn projects_column_for(sql: &str, measure: &str) -> bool { + sql.contains(&format!(r#""{}""#, measure.replace('.', "__"))) +} + +fn expect_no_own_aggregate_error(ctx: &TestContext, query: &str, measure: &str) { + let result = ctx.build_sql(query); + assert!(result.is_err(), "expected the query to be refused"); + let err_msg = result.unwrap_err().to_string(); + assert!( + err_msg.contains("has no aggregate of its own"), + "Error should explain the missing aggregate, got: {err_msg}" + ); + assert!( + err_msg.contains(measure), + "Error should name {measure}, got: {err_msg}" + ); +} + +/// A calculated measure alone: its components are aggregated over the +/// deduplicated key set, so the fan-out on `p1` does not skew the ratio. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_alone_by_fan_out_dimension() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.success_rate + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 1, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// The calculated measure's components requested directly, to pin down the +/// numbers the ratio is built from. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_components_by_fan_out_dimension() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.success_count + - payments.count + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 0, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// A measure that needs the `rates` join next to one that does not. Both are on +/// `payments`, and `count_distinct` keeps its own value correct under the +/// fan-out while `converted_value` is deduplicated through the keys subquery. +#[tokio::test(flavor = "multi_thread")] +async fn test_joined_measure_with_distinct_count_by_fan_out_dimension() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.converted_value + - payments.count + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 1, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// Two plain `sum` measures of the same cube, one of which needs a wider join. +/// No calculated measure involved - this is the join grouping on its own. +#[tokio::test(flavor = "multi_thread")] +async fn test_two_sums_of_one_cube_needing_different_joins() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.total_amount + - payments.converted_value + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 2, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// A calculated measure alongside a measure that needs a wider join. Both sit on +/// `payments`, so they share a key cube, but they need different join trees and +/// therefore land in separate subqueries. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_with_joined_measure() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.success_rate + - payments.converted_value + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 2, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// Same shape, but the calculated measure is built from `sum` components, which +/// do not survive row multiplication on their own. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_over_sums_with_joined_measure() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.success_amount_rate + - payments.converted_value + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 2, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// Three measures of one cube needing three nested join trees: `{payments}`, +/// `{payments, rates}` and `{payments, rates, merchants}`. +#[tokio::test(flavor = "multi_thread")] +async fn test_three_nested_join_trees_of_one_cube() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.total_amount + - payments.converted_value + - payments.net_value + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 3, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// Two lookup cubes that are siblings rather than nested: `{payments, rates}` +/// against `{payments, merchants}`, neither containing the other. +#[tokio::test(flavor = "multi_thread")] +async fn test_two_sibling_lookup_join_trees_of_one_cube() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.converted_value + - payments.commissioned_value + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 2, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// A calculated measure that reaches another cube, on its own. Reaching it +/// forces the measure subquery, which strips the aggregates its components +/// carry and leaves the expression with nothing to re-apply. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_reaching_other_cubes_alone() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.rate_vs_commission + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + expect_no_own_aggregate_error(&ctx, query, "payments.rate_vs_commission"); +} + +/// The same measure next to one that needs neither of the cubes it reaches, so +/// the two land on separate join trees. The calculated leg is refused all the +/// same - splitting the query does not give the expression an aggregate. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_over_components_with_different_joins() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.rate_vs_commission + - payments.total_amount + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + expect_no_own_aggregate_error(&ctx, query, "payments.rate_vs_commission"); +} + +/// A star: the dimension pulls in one one-to-many branch and the measure pulls +/// in another. A `sum` on `payments` filtered by a `payment_tags` value has no +/// well-defined value once a payment can carry several tags, so it is refused +/// rather than silently double-counted. +#[tokio::test(flavor = "multi_thread")] +async fn test_star_with_two_fan_out_branches_is_rejected() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.total_amount + - payments.tagged_amount + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let result = ctx.build_sql(query); + assert!( + result.is_err(), + "A measure reaching a second fan-out branch should not plan" + ); + let err_msg = result.unwrap_err().to_string(); + assert!( + err_msg.contains("lead to row multiplication"), + "Error should report row multiplication, got: {err_msg}" + ); + assert!( + err_msg.contains("payments.tagged_amount"), + "Error should name the offending measure, got: {err_msg}" + ); +} + +/// Measures owned by two sibling fan-out cubes plus the cube they hang off. Each +/// gets its own leaf query joined by `FullKeyAggregate`, but none of them is +/// multiplied - the two fan-out cubes sit on the `many` side of their own joins, +/// and `total_amount`'s tree stops at `{payments}` because the dimension does - +/// so nothing takes the keys path. +#[tokio::test(flavor = "multi_thread")] +async fn test_measures_of_two_sibling_fan_out_cubes() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payment_meta.count + - payment_tags.count + - payments.total_amount + dimensions: + - payments.status + order: + - id: payments.status + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 0, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// Several join-tree shapes at once: a bare aggregate on `{payments}`, a +/// three-cube measure on `{payments, rates, merchants}`, and a calculated +/// measure whose components stay inside `{payments}`. Two legs, not three - the +/// grouping is by join tree, so the calculated measure shares one with the bare +/// aggregate rather than getting its own. +#[tokio::test(flavor = "multi_thread")] +async fn test_several_join_tree_shapes_in_one_query() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.total_amount + - payments.net_value + - payments.success_rate + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 2, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// A calculated measure whose dependencies are all measures, but whose sql also +/// reads its own cube's table. Letting the component leave would strand that +/// read in a select the cube is not joined into. Paired with a measure of +/// another cube so the query is planned through classification, and grouped by a +/// dimension that multiplies nothing, so the measure renders whole and the +/// result is checkable. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_reading_its_own_cube_directly() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.converted_per_max_amount + - payment_meta.count + dimensions: + - payments.status + order: + - id: payments.status + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 0, "sql: {sql}"); + assert!( + projects_column_for(&sql, "payments.converted_per_max_amount"), + "sql: {sql}" + ); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// A calculated measure reaching another cube past its component measure, so the +/// aggregate around `MAX({rates.fx_rate})` is the measure's own. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_reaching_past_its_components() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.amount_over_fx + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + expect_no_own_aggregate_error(&ctx, query, "payments.amount_over_fx"); +} + +/// A calculated measure whose components resolve to join trees rooted at +/// different cubes. Evaluating it in one place and splitting it apart give +/// different answers, so it is refused rather than answered either way. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_over_components_rooted_at_different_cubes() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.gold_share + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + expect_no_own_aggregate_error(&ctx, query, "payments.gold_share"); +} + +/// A dimension of the joined cube puts the calculated measure and the one that +/// reaches into the same group, so the measure subquery is built for a measure +/// that never asked for it. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_pulled_into_a_shared_measure_subquery() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.success_rate + - payments.converted_value + dimensions: + - payment_meta.value + - rates.currency + "}; + + expect_no_own_aggregate_error(&ctx, query, "payments.success_rate"); +} + +/// The same shape a rollup stores. The measure subquery is never rendered, so +/// the query is answered rather than refused - which is why the check belongs +/// where the subquery is built and not where the plan is. +#[tokio::test(flavor = "multi_thread")] +async fn test_refused_shape_is_answered_from_a_rollup() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.max_fx_rate + dimensions: + - payment_meta.value + "}; + + let (_sql, usages) = ctx.build_sql_with_used_pre_aggregations(query).unwrap(); + let names: Vec<&str> = usages.iter().map(|u| u.name().as_str()).collect(); + assert_eq!(names, vec!["max_fx_by_meta_value"]); +} + +/// A calculated measure reaching another cube with no component measure at all - +/// its aggregate is written by hand over the joined cube's column. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_without_component_measures() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.max_fx_rate + - payments.total_amount + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + expect_no_own_aggregate_error(&ctx, query, "payments.max_fx_rate"); +} + +/// A calculated measure that reaches another cube but is not multiplied. It is +/// read off a leaf-measure query that keeps its aggregate, so it needs no +/// decomposition and must be evaluated whole: its components root at different +/// cubes, and splitting them would divide `gold_amount` by a `total_amount` +/// taken over a different set of rows than its own leg sees. +/// +/// Evaluated whole, both sides of the ratio see the `customers`-joined rows: +/// `SUCCESS` is 100 * 300 / 700, p5 having no customer, while the sibling +/// `payments.total_amount` column reads 1200 over all rows. The two columns are +/// not meant to reconcile - the ratio is taken within its own leg. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_reaching_other_cubes_without_multiplication() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.gold_share + - payments.total_amount + dimensions: + - payments.status + order: + - id: payments.status + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 0, "sql: {sql}"); + assert!( + projects_column_for(&sql, "payments.gold_share"), + "sql: {sql}" + ); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// Split groups whose trees root at different cubes: `total_amount` roots at +/// `payments`, while `gold_amount` reaches `customers`, which owns the join and +/// therefore becomes the root. Customerless payments are unreachable from the +/// second root, so that leg reports only `A` and `B` while the first also +/// reports `C`; the result must carry `C` with a null `gold_amount`. +#[tokio::test(flavor = "multi_thread")] +async fn test_split_groups_rooted_at_different_cubes() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.total_amount + - payments.gold_amount + dimensions: + - payment_meta.value + order: + - id: payment_meta.value + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 2, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} + +/// Without the fan-out dimension there is a single join group, so the same pair +/// of measures plans and runs. +#[tokio::test(flavor = "multi_thread")] +async fn test_calculated_measure_with_joined_measure_without_fan_out() { + let ctx = create_context(); + + let query = indoc! {" + measures: + - payments.success_rate + - payments.converted_value + dimensions: + - payments.status + order: + - id: payments.status + "}; + + let sql = ctx.build_sql(query).unwrap(); + assert_eq!(keys_subquery_count(&sql), 0, "sql: {sql}"); + + if let Some(result) = ctx.try_execute_pg(query, SEED).await { + insta::assert_snapshot!(result); + } +} diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/mod.rs b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/mod.rs index eb3176e488533..c2c09b60c98de 100644 --- a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/mod.rs +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/mod.rs @@ -1,6 +1,7 @@ mod advanced_features; mod advanced_filters; mod calc_groups; +mod calculated_multi_fact; mod calendar; mod chained_subquery; mod combinations; diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_alone_by_fan_out_dimension.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_alone_by_fan_out_dimension.snap new file mode 100644 index 0000000000000..b769ee4cf8eaf --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_alone_by_fan_out_dimension.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__success_rate +--------------------+----------------------- +A | 66.6666666666666667 +B | 100.0000000000000000 +C | 100.0000000000000000 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_components_by_fan_out_dimension.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_components_by_fan_out_dimension.snap new file mode 100644 index 0000000000000..7dc6bc1be1855 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_components_by_fan_out_dimension.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__success_count | payments__count +--------------------+-------------------------+---------------- +A | 2 | 3 +B | 1 | 1 +C | 1 | 1 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_over_sums_with_joined_measure.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_over_sums_with_joined_measure.snap new file mode 100644 index 0000000000000..49258fd735ad8 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_over_sums_with_joined_measure.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__success_amount_rate | payments__converted_value +--------------------+-------------------------------+-------------------------- +A | 50.0000000000000000 | 600.0000000000000000 +B | 100.0000000000000000 | 200.0000000000000000 +C | 100.0000000000000000 | 500.0000000000000000 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_reaching_other_cubes_without_multiplication.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_reaching_other_cubes_without_multiplication.snap new file mode 100644 index 0000000000000..f3858f2bb1952 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_reaching_other_cubes_without_multiplication.snap @@ -0,0 +1,8 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payments__status | payments__gold_share | payments__total_amount +-----------------+----------------------+----------------------- +DECLINED | NULL | 300.00 +SUCCESS | 42.8571428571428571 | 1200.00 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_reading_its_own_cube_directly.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_reading_its_own_cube_directly.snap new file mode 100644 index 0000000000000..ca64e1c34ae1a --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_reading_its_own_cube_directly.snap @@ -0,0 +1,8 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payments__status | payments__converted_per_max_amount | payment_meta__count +-----------------+------------------------------------+-------------------- +DECLINED | 1.00000000000000000000 | 1 +SUCCESS | 2.0000000000000000 | 5 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_with_joined_measure.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_with_joined_measure.snap new file mode 100644 index 0000000000000..dca143f69ccab --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_with_joined_measure.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__success_rate | payments__converted_value +--------------------+------------------------+-------------------------- +A | 66.6666666666666667 | 600.0000000000000000 +B | 100.0000000000000000 | 200.0000000000000000 +C | 100.0000000000000000 | 500.0000000000000000 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_with_joined_measure_without_fan_out.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_with_joined_measure_without_fan_out.snap new file mode 100644 index 0000000000000..9a3e7a477ee50 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__calculated_measure_with_joined_measure_without_fan_out.snap @@ -0,0 +1,8 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payments__status | payments__success_rate | payments__converted_value +-----------------+------------------------+-------------------------- +DECLINED | 0.00000000000000000000 | 300.0000000000000000 +SUCCESS | 100.0000000000000000 | 1000.0000000000000000 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__joined_measure_with_distinct_count_by_fan_out_dimension.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__joined_measure_with_distinct_count_by_fan_out_dimension.snap new file mode 100644 index 0000000000000..5c260e187d874 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__joined_measure_with_distinct_count_by_fan_out_dimension.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__converted_value | payments__count +--------------------+---------------------------+---------------- +A | 600.0000000000000000 | 3 +B | 200.0000000000000000 | 1 +C | 500.0000000000000000 | 1 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__measures_of_two_sibling_fan_out_cubes.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__measures_of_two_sibling_fan_out_cubes.snap new file mode 100644 index 0000000000000..51e45f25c2d57 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__measures_of_two_sibling_fan_out_cubes.snap @@ -0,0 +1,8 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payments__status | payment_meta__count | payment_tags__count | payments__total_amount +-----------------+---------------------+---------------------+----------------------- +DECLINED | 1 | 0 | 300.00 +SUCCESS | 5 | 4 | 1200.00 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__several_join_tree_shapes_in_one_query.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__several_join_tree_shapes_in_one_query.snap new file mode 100644 index 0000000000000..1aa2867c5ee41 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__several_join_tree_shapes_in_one_query.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__total_amount | payments__net_value | payments__success_rate +--------------------+------------------------+--------------------------+----------------------- +A | 600.00 | 180.00000000000000000000 | 66.6666666666666667 +B | 400.00 | 100.00000000000000000000 | 100.0000000000000000 +C | 500.00 | 50.00000000000000000000 | 100.0000000000000000 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__split_groups_rooted_at_different_cubes.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__split_groups_rooted_at_different_cubes.snap new file mode 100644 index 0000000000000..fd85f1a7bb782 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__split_groups_rooted_at_different_cubes.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__total_amount | payments__gold_amount +--------------------+------------------------+---------------------- +A | 600.00 | 300.00 +B | 400.00 | NULL +C | 500.00 | NULL diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__three_nested_join_trees_of_one_cube.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__three_nested_join_trees_of_one_cube.snap new file mode 100644 index 0000000000000..9e063f4d1862c --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__three_nested_join_trees_of_one_cube.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__total_amount | payments__converted_value | payments__net_value +--------------------+------------------------+---------------------------+------------------------- +A | 600.00 | 600.0000000000000000 | 180.00000000000000000000 +B | 400.00 | 200.0000000000000000 | 100.00000000000000000000 +C | 500.00 | 500.0000000000000000 | 50.00000000000000000000 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__two_sibling_lookup_join_trees_of_one_cube.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__two_sibling_lookup_join_trees_of_one_cube.snap new file mode 100644 index 0000000000000..e1b188c787161 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__two_sibling_lookup_join_trees_of_one_cube.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__converted_value | payments__commissioned_value +--------------------+---------------------------+----------------------------- +A | 600.0000000000000000 | 180.000000 +B | 200.0000000000000000 | 200.000000 +C | 500.0000000000000000 | 50.000000 diff --git a/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__two_sums_of_one_cube_needing_different_joins.snap b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__two_sums_of_one_cube_needing_different_joins.snap new file mode 100644 index 0000000000000..3273779567d32 --- /dev/null +++ b/rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/snapshots/cubesqlplanner__tests__integration__calculated_multi_fact__two_sums_of_one_cube_needing_different_joins.snap @@ -0,0 +1,9 @@ +--- +source: cubesqlplanner/cubesqlplanner/src/tests/integration/calculated_multi_fact.rs +expression: result +--- +payment_meta__value | payments__total_amount | payments__converted_value +--------------------+------------------------+-------------------------- +A | 600.00 | 600.0000000000000000 +B | 400.00 | 200.0000000000000000 +C | 500.00 | 500.0000000000000000