Skip to content

SCRUM-1068-explore profile flags a boolean-shaped column with >2 values - #326

Merged
marcociav-exmergo merged 2 commits into
mainfrom
SCRUM-1068-issue-218
Aug 19, 2026
Merged

SCRUM-1068-explore profile flags a boolean-shaped column with >2 values#326
marcociav-exmergo merged 2 commits into
mainfrom
SCRUM-1068-issue-218

Conversation

@emnarahmouni-exmergo

Copy link
Copy Markdown
Contributor

Closes #218.
A column named like a two-valued flag (is_, has_, *_flag, *_yn, *_ind) can hold more than two distinct non-null values, a mixed-encoding defect that went unreported: raw_workspaces.primary_ws_yn on the public ADE-bench helixops_saas.duckdb holds five values under a name that promises two.
explore profile now emits a data-quality observation when a boolean-ish name and a non-null distinct count above two coincide.
Where the value domain (#203) is already known for that column, the observation names each encoding with its count; where it's not (the column failed one of that feature's own eligibility gates), it still fires on the count alone, since not being able to name the values is not evidence there are only two of them.
A genuinely BOOLEAN-typed column is excluded outright, whatever its name it cannot hold more than two values by construction.
Deliberately does not try to distinguish a data-quality defect from a genuine third state; the tool can't make that call, so it names what it found and lets the caller decide (a required acceptance criterion, not an oversight).

image
  1. primary_ws_yn (5 values) → flagged, naming them: "primary_ws_yn looks boolean-shaped by name but holds 5 distinct non-null values: 1=20, N=20, No=20, Y=20, Yes=20; a two-valued read of it will misclassify some rows" every encoding named with its exact count.
  2. has_discount (clean 2-value flag) → silent: no entry for it anywhere in data_quality.
  3. onboarding_stage_flag (3 values, a genuine third state) → still flagged: "...holds 3 distinct non-null values: new=34, active=33, churned=33..." fires exactly as the acceptance criteria requires, since the tool can't (and isn't asked to) tell a defect apart from a real third state.
  4. is_native_bool (real BOOLEAN type) → silent: no entry, even though nothing else about its name would exclude it.

@marcociav-exmergo marcociav-exmergo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, thank you @emnarahmouni-exmergo

@marcociav-exmergo
marcociav-exmergo merged commit 04c5b39 into main Aug 19, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

explore profile: flag a boolean-shaped flag with more than two values

2 participants