SCRUM-1069-explore profile: detect cumulative and snapshot measures - #327
Merged
Conversation
marcociav-exmergo
approved these changes
Aug 19, 2026
marcociav-exmergo
left a comment
Member
There was a problem hiding this comment.
Looks good, thanks @emnarahmouni-exmergo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes : #219
explore profile --check-cumulative(#219): flags a numeric column thatlooks like a running total, an account balance, or a point-in-time
snapshot instead of a per-row increment. This shape profiles identically
to an ordinary measure (same type, null fraction, uniqueness), so summing
it downstream is a common and severely damaging misreading that nothing
else in
explore profilecatches.How it works
itself the table's key) ordered by a temporal column, a cumulative or
snapshot measure almost never decreases from one observation to the
next. A genuine per-row increment has natural ups and downs.
once in DuckDB SQL and transpiled per connector dialect via sqlglot
(mirrors
relationships.py's overlap-probe pattern). No row's measurevalue ever leaves the engine, only comparison counts.
--verify: the base profile always completesand saves first; the check runs as a second, individually skippable
phase, dry-run priced only once profiling knows what to test, and can
come back as
pending_confirmationon a billed connector rather thandiscarding already-paid-for work.
explicit note, never silently reported as clean.
Bugs found and fixed during manual testing
Two real bugs surfaced while exercising this against a live DuckDB file
(not caught by the initial test pass, since it always exercised a fresh
scan):
object the cache write carries forward, so a note appended only to it
showed in that run's output but never reached disk. Fixed by resolving
the cache's own object by identifier before appending.
same sentence. Fixed by making the append idempotent (skip if the exact
text is already present).
Both are covered by dedicated regression tests, plus manual stress
testing across multiple objects in one call,
--refreshcombined withthe flag, a first-ever call already using the flag, and a table with no
eligible shape.