Skip to content

SCRUM-1069-explore profile: detect cumulative and snapshot measures - #327

Merged
marcociav-exmergo merged 2 commits into
mainfrom
SCRUM-1069-issue-219
Aug 19, 2026
Merged

SCRUM-1069-explore profile: detect cumulative and snapshot measures#327
marcociav-exmergo merged 2 commits into
mainfrom
SCRUM-1069-issue-219

Conversation

@emnarahmouni-exmergo

Copy link
Copy Markdown
Contributor

Closes : #219
explore profile --check-cumulative (#219): flags a numeric column that
looks like a running total, an account balance, or a point-in-time
snapshot instead of a per-row increment. This shape profiles identically
to an ordinary measure (same type, null fraction, uniqueness), so summing
it downstream is a common and severely damaging misreading that nothing
else in explore profile catches.

How it works

  • Signal: within an entity (a repeating, id-shaped column that is not
    itself the table's key) ordered by a temporal column, a cumulative or
    snapshot measure almost never decreases from one observation to the
    next. A genuine per-row increment has natural ups and downs.
  • The probe is a single aggregate-only, window-function query authored
    once in DuckDB SQL and transpiled per connector dialect via sqlglot
    (mirrors relationships.py's overlap-probe pattern). No row's measure
    value ever leaves the engine, only comparison counts.
  • Priced and gated like --verify: the base profile always completes
    and saves first; the check runs as a second, individually skippable
    phase, dry-run priced only once profiling knows what to test, and can
    come back as pending_confirmation on a billed connector rather than
    discarding already-paid-for work.
  • A table missing an entity key or a temporal column is skipped with an
    explicit note, never silently reported as clean.

Bugs found and fixed during manual testing

Two real bugs surfaced while exercising this against a live DuckDB file
(not caught by the initial test pass, since it always exercised a fresh
scan):

  • A dataset served from the freshness cache is a deep copy, not the
    object the cache write carries forward, so a note appended only to it
    showed in that run's output but never reached disk. Fixed by resolving
    the cache's own object by identifier before appending.
  • Running the check again on an already-flagged dataset duplicated the
    same sentence. Fixed by making the append idempotent (skip if the exact
    text is already present).

Both are covered by dedicated regression tests, plus manual stress
testing across multiple objects in one call, --refresh combined with
the flag, a first-ever call already using the flag, and a table with no
eligible shape.

image balance correctly flagged as cumulative, purely from its shape in the data.

@marcociav-exmergo marcociav-exmergo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks @emnarahmouni-exmergo

@marcociav-exmergo
marcociav-exmergo merged commit 99fff5d into main Aug 19, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

explore profile: detect cumulative and snapshot measures

2 participants