Is your feature request related to a problem? Please describe.
When evaluating machine learning models trained on biological datasets, users have no way to verify whether their train and test sets are free of data leakage within BioProfileKit. A test set that contains sequences, peptides, or samples already seen during training will produce artificially inflated performance metrics — a critical issue in immunological and genomic benchmarking where this problem is well-documented but frequently overlooked.
Describe the solution you'd like
Extend the CLI to accept an optional second input file representing the test set:
bioprofilekit -i train.csv --test test.csv
When --test is provided, BioProfileKit performs a dedicated leakage analysis between the two datasets:
- Exact row leakage: Count rows in the test set that are identical to any row in the train set (full-row match after normalisation)
- Exact sequence leakage: For any detected sequence column (DNA, RNA, protein), count sequences in the test set that appear verbatim in the train set, broken down by sequence length bucket
- Near-duplicate leakage (optional, activated via
--homology): Flag test sequences with ≥ 90 % identity to a train sequence using CD-HIT or Biopython fallback
- Feature-level overlap: For categorical columns (organism, epitope type, MHC allele), report the fraction of test-set categories that also appear in the train set — values of 100 % indicate no novel categories in the test set, which may be intentional but should be visible
- Target distribution shift: If a target column is set via
--target-column, compare its distribution between train and test using a KS-test (numeric) or chi-squared test (categorical) and flag significant divergence
Results are surfaced as a dedicated Train/Test Leakage section in general_statistics.jinja and as quality checks in the Quality Assessment Report (fail if exact row leakage > 5 %, warn if > 1 %).
Describe alternatives you've considered
Detecting leakage risk from a single dataset based on duplicate row counts alone (the previous formulation of this issue). This is a useful heuristic but cannot confirm actual leakage without access to both splits. Users who have already created their splits deserve a definitive answer rather than an estimate.
Additional context
The two-file workflow maps cleanly onto the existing architecture: read_file() is called twice, producing two DataFrames. Column matching between train and test should be validated upfront (warn if schemas differ). The leakage report should be separate from the main column-level analysis so that the report structure remains coherent when only a single file is provided. For the de.NBI web deployment, this feature would map to a two-file upload interface on the landing page.
Is your feature request related to a problem? Please describe.
When evaluating machine learning models trained on biological datasets, users have no way to verify whether their train and test sets are free of data leakage within BioProfileKit. A test set that contains sequences, peptides, or samples already seen during training will produce artificially inflated performance metrics — a critical issue in immunological and genomic benchmarking where this problem is well-documented but frequently overlooked.
Describe the solution you'd like
Extend the CLI to accept an optional second input file representing the test set:
bioprofilekit -i train.csv --test test.csvWhen
--testis provided, BioProfileKit performs a dedicated leakage analysis between the two datasets:--homology): Flag test sequences with ≥ 90 % identity to a train sequence using CD-HIT or Biopython fallback--target-column, compare its distribution between train and test using a KS-test (numeric) or chi-squared test (categorical) and flag significant divergenceResults are surfaced as a dedicated Train/Test Leakage section in
general_statistics.jinjaand as quality checks in the Quality Assessment Report (failif exact row leakage > 5 %,warnif > 1 %).Describe alternatives you've considered
Detecting leakage risk from a single dataset based on duplicate row counts alone (the previous formulation of this issue). This is a useful heuristic but cannot confirm actual leakage without access to both splits. Users who have already created their splits deserve a definitive answer rather than an estimate.
Additional context
The two-file workflow maps cleanly onto the existing architecture:
read_file()is called twice, producing two DataFrames. Column matching between train and test should be validated upfront (warn if schemas differ). The leakage report should be separate from the main column-level analysis so that the report structure remains coherent when only a single file is provided. For the de.NBI web deployment, this feature would map to a two-file upload interface on the landing page.