Is your feature request related to a problem? Please describe.
Biological datasets frequently contain inconsistent spellings of the same entity — E.coli, E. coli, e.coli, Ecoli — that inflate cardinality, break grouping operations, and cause false negatives in taxonomy validation. These are invisible to standard value-count analysis.
Describe the solution you'd like
Add a check_potential_typos() function in data_quality.py that:
- For each categorical column with fewer than 500 unique values, compute pairwise edit distances (Levenshtein) between unique values using
rapidfuzz or difflib.SequenceMatcher
- Flag pairs with similarity ≥ 0.85 and count ≤ 5 % of the dominant variant as potential typos
- Optionally, cross-reference against the existing taxonomy vocabulary (if
--tax is active) to distinguish genuine synonyms from typos
- Return a list of
(suspect_value, likely_intended, similarity_score) tuples
Surface in columns.jinja as a "Potential Typos" sub-tab with a searchable table of flagged pairs.
Describe alternatives you've considered
Phonetic matching (Soundex, Metaphone) to catch pronunciation-based typos. Less relevant for Latin binomial nomenclature but worth exploring for common-name fields.
Additional context
rapidfuzz is significantly faster than pure-Python Levenshtein and is already used by many bioinformatics tools. For large unique-value sets, restrict to a sample of the most frequent values to keep runtime manageable.
Is your feature request related to a problem? Please describe.
Biological datasets frequently contain inconsistent spellings of the same entity —
E.coli,E. coli,e.coli,Ecoli— that inflate cardinality, break grouping operations, and cause false negatives in taxonomy validation. These are invisible to standard value-count analysis.Describe the solution you'd like
Add a
check_potential_typos()function indata_quality.pythat:rapidfuzzordifflib.SequenceMatcher--taxis active) to distinguish genuine synonyms from typos(suspect_value, likely_intended, similarity_score)tuplesSurface in
columns.jinjaas a "Potential Typos" sub-tab with a searchable table of flagged pairs.Describe alternatives you've considered
Phonetic matching (Soundex, Metaphone) to catch pronunciation-based typos. Less relevant for Latin binomial nomenclature but worth exploring for common-name fields.
Additional context
rapidfuzzis significantly faster than pure-Python Levenshtein and is already used by many bioinformatics tools. For large unique-value sets, restrict to a sample of the most frequent values to keep runtime manageable.