Merge Development into Main --> Release: BioProfileKit CLI v1.0

[hansen-maria]

Branch: development → main
Type: Feature / Major Version Release
Authors: Maria Hansen, Sonja Diedrich, Julian Hahnfeld, Felix Knopp

---

Overview

This PR merges the complete development history of BioProfileKit into main and marks the v1.0 release of the CLI tool.

BioProfileKit is a domain-specific automated data profiling tool for bioinformatics tabular data (CSV, TSV, JSON, Parquet). It generates interactive HTML reports covering dataset-level and column-level exploratory data analysis, biological sequence profiling (DNA/RNA, protein), biological metadata validation (taxonomy, functional annotation, lab measurements), and multivariate statistics — all from a single CLI invocation.

---

What's Included

Core Analysis Pipeline (src/)

File I/O (data_utils/)

• file_reader.py — unified reader for CSV, TSV, JSON, and Parquet with automatic delimiter detection via csv.Sniffer, header inference, and index-column behaviour consistent across formats
• parse_parquet via pyarrow with index-column parity to CSV reader
• Edge case handling: single-column CSV parsing, JSON index errors, empty float64 columns re-cast to StringDtype for correct downstream categorical treatment (Pandas 3.x compatibility)
• remote_data.py — NCBI taxonomy ID download

Column Analysis (analysis/)

• overview.py — GeneralOverview and ColumnOverview dataclasses; taxonomy_candidate heuristic using column-name keywords and binomial nomenclature pattern matching to show Not-Checked badges only for plausible taxonomy columns
• numeric_analysis.py — descriptive statistics, quantiles, MAD, CoV, skewness, kurtosis, infinity check, zero/negative counts, adjusted IQR outlier detection, Grubbs/modified Z-score support
• categorical_analysis.py — unique categories, entropy, Gini, Simpson diversity, cardinality ratio, effective cardinality, CIB ratio, rare category detection (≤5% threshold), top-N value counts; guards for empty Series in top_1_coverage and cib_ratio
• multivariate_analysis.py (renamed from multivariate.py) — Pearson, Cramér's V, η² correlation matrices; top_associations ranked list; feature–target correlation and plot; mutual information ranking and relationship plots; Little's MCAR test; missing value matrix and bar chart; balance plot; box plots; scatter matrix; autosize enabled on all Plotly figures for responsive layout

Biological Analysis (biological/)

• sequence_data.py — DNA/RNA: GC content, GC/AT skew, CpG/TpA O/E ratio, ambiguous base ratio, codon completeness, low complexity, reverse complement ratio, k-mer frequencies; Protein: GRAVY, cysteine count, disorder propensity, stop codon ratio, ambiguous residue ratio, length distribution, amino acid group distribution, k-mer frequencies; WebLogo sequence logo (SVG fragment extraction via regex — prevents Bootstrap 3 injection from WebLogo's full-HTML output)
• sequence_detection.py — heuristic classifier for DNA, RNA, protein, and non-sequence columns; guard for k-mer length > sequence length
• taxonomy.py — NCBI taxonomy validation; rank distribution; outdated name detection; invalid entry collection; mixed-taxonomy detection
• functional_annotation.py — COG and GO term validation against downloaded reference data
• measurement_data.py (Julian Hahnfeld) — unit detection via regex; measurement column classification; unit count and mixed-unit detection; refactored for speed and memory efficiency

Quality Assessment (quality_assessment/)

• quality_assessment.py — 17 checks across four categories:
  • Dataset Structure: sample size vs. feature count, duplicate rows, duplicate columns, empty rows
  • Column Quality: missing values, feature variance, high cardinality (≥90% unique), mixed types, infinity, skewness, outliers, empty columns
  • Biological: sequence validity, reverse complement redundancy, taxonomy validity, unit validity
  • Relationships: target leakage, multicollinearity (>0.90 association)
• Pass / Warn / Fail status per check and per category; overall banner
• detail_link resolution mapping quality checks to the relevant report page

CLI (cli/)

• app.py — Click-based CLI entry point; orchestrates the full pipeline; collects Click context parameters and writes both HTML report and results.json on success
• report_writer.py — Jinja2-based HTML renderer; writes index.html, numeric_data.html, columns.html, general_statistics.html; copies static assets; resolves quality check detail links
• report_json.py (new) — machine-readable JSON output alongside the HTML report; schema-versioned; numpy-safe serialisation with _num() and _json_default; per-column blocks for numeric, categorical, sequence, taxonomy, measurement, and cross-column relations
• utils.py — print_step, info, _fmt_duration CLI helpers
• Makefile — build and run targets

Cython Extensions

• wrapper_utils.pyx — core utility wrapper
• taxonomy_validator.pyx — fast taxonomy name/ID lookup

---

HTML Report Templates (templates/)

All four Jinja2 templates were built from scratch and iteratively expanded:

LandingPage.jinja

• Logo, tagline, call-to-action card
• Navigation cards linking to the three analysis sub-pages
• Flat navbar (no dropdown); colorblind toggle as fixed bottom-right pill (persisted via localStorage; DOMContentLoaded listener prevents IIFE-before-element race condition)

numeric_overview.jinja (Analysis Overview)

• Quality Assessment Report — overall status banner (PASS / MINOR / MAJOR); category cards with pass/warn/fail indicators; messages rendered as lists; detail links to relevant sub-pages; quality parameter optional for backwards compatibility with older tests
• Dataset Overview & Column Types — side-by-side tables
• Conditional Duplicates section — hidden when no duplicates present
• Taxonomy Not-Checked summary moved to overview level

general_statistics.jinja (Multivariate Statistics)

• Four named sections with section-filter buttons: Missingness, Correlations, Target Analysis, General Statistics
• Correlation Plots — four-tab layout: Association Heatmap, Association Table (DataTables), Pearson heatmap, Cramér's V heatmap, η² heatmap
• Mutual Information — ranked table and relationship plots in tabs
• Feature–Target — correlation chart and balance plot (conditional on target_column)
• MCAR — Little's test result tab
• Plotly resize fix: shown.bs.tab event triggers Plotly.relayout on .plotly-graph-div elements within the activated tab panel
• High-dimensional rendering warning with cloud service link

columns.jinja (Column Analysis)

• Collapsible filter bar with text search, column type filter, and quality flag toggles (pass/warn/fail/any-alert); cbf-active CSS class toggle
• Per-column accordion cards with:
  • Numeric: descriptive statistics, quantiles, outlier detection results, numeric quality metrics
  • Categorical: value distribution, top-N counts, cardinality metrics, imbalance and rarity metrics
  • DNA/RNA: GC content, AT/GC skew, CpG/TpA O/E, ambiguous base ratio, codon completeness, low complexity, reverse complement ratio, k-mer frequency bar chart, sequence logo (SVG); per-sequence and column-wide metrics separated
  • Protein: GRAVY, cysteine count, disorder propensity, stop codon ratio, ambiguous residue ratio, length distribution tab, amino acid group distribution, k-mer frequency chart, sequence logo; protein features tab
  • Taxonomy: rank distribution sub-tabs; invalid entries table; outdated names table; tax-ID sub-tab; three-state badges (Valid / Invalid / Not Checked) shown only on taxonomy candidate columns
  • Functional Annotation: COG/GO validation badges
  • Measurement: unit detection results, unit count breakdown
• DataTables integration with consistent typography
• Pagination of toggle elements beyond 20 columns
• Accessibility: ARIA attributes on interactive elements
• Colorblind-safe theme (Okabe-Ito palette) with data-cb-theme switching; WCAG AA contrast verified across both themes
• localStorage persistence for colorblind preference across pages

---

Static Assets (static/)

• cb.css — BioProfileKit color scheme (BPK-Pink #932263, BPK-Dark-Grey #3D4756); colorblind theme (data-cb-theme=colorblind); button, badge, pagination, nav-tab, and search input overrides; fixed footer layout
• bootstrap.css — Bootstrap 5.3.3
• GitHub SVG icon; favicon

---

Test Suite (tests/)

Unit Tests (tests/unit/)

Module	Tests
categorical_analysis.py	value counts, entropy, Gini, cardinality metrics, empty Series guards
numeric_analysis.py	descriptive stats, outlier detection, infinity/zero counts
outlier_detection.py	adjusted IQR bounds, medcouple, Z-score methods
data_quality.py	all 17 quality checks, pass/warn/fail thresholds
functional_annotation.py	COG/GO validation, unknown ID handling
sequence_data.py	GC content, k-mer counting, WebLogo SVG extraction, nucleotide type checks
sequence_detection.py	DNA/RNA/protein/non-sequence classification
file_reader.py	CSV/TSV/JSON/Parquet reading, sniffer edge cases, empty column re-cast
multivariate_analysis.py	correlation matrices, top associations, MCAR

Integration Tests (tests/integration/)

Module	Tests
test_app.py	CLI invocation (exit codes, output format, parameter passing), file format support (CSV/TSV/Parquet), sequence column detection, duplicate detection, edge cases (all-null column, single row, all-numeric, all-categorical)
test_report_writer.py	output directory creation, all four HTML files generated, template context passing, autoescaping, _render_to_file unit tests

Coverage: ≥70% enforced in CI
Total: 434 tests — 0 failed

CI/CD (.github/workflows/)

• GitHub Actions pipeline: install dependencies, Cython build step, pytest with coverage report, coverage threshold enforcement

---

Infrastructure

• pyproject.toml — dependencies, build configuration, coverage settings, minimum version bumps
• Dockerfile — base image for CLI tool (foundation for backend image)
• compose.yaml / docker-compose.yml — local development stack
• Makefile — build, test, and run targets
• MANIFEST.in — package data inclusion for importlib_resources
• requirements.txt — pinned dependencies with minimum version bumps
• CODE_OF_CONDUCT.md, README.md — project documentation and badges
• .gitignore — updated for build artifacts and test output

---

Breaking Changes

None. This is the initial public release (version 1.0); there is no prior stable BPK.

---

Known Limitations & Open Issues

The following items are tracked as GitHub Issues and deferred to v1.1:

#	Category	Topic
1	General	Alert count summary in dataset overview
2	General	Report export (e.g. PDF)
3	Column	ID column detection and exclusion heuristic
4	Column	Whitespace issue detection and flagging
5	Column	Encoding issues (non-ASCII / mojibake) detection
6	Numeric	Normality tests (Shapiro-Wilk, Anderson-Darling)
7	Numeric	Distribution fitting
8	Categorical	Potential typo detection (fuzzy near-duplicate values)
9	DNA/RNA	Duplicate sequence detection
10	DNA/RNA	Sequence homology clustering
11	Protein	Duplicate sequence detection
12	Protein	Homology clustering
13	Biological	UniProt ID validation
14	Multivariate	Full missing value pattern analysis (MCAR / MAR / MNAR)
15	Multivariate	Train/test leakage risk detection via dual-file upload
16	Multivariate	Cross-column ID consistency checks
17	Metadata	Database dump export
18	Jinja	Plotly JSON rendering refactor (decouple data from HTML)

---

Checklist

• All 434 tests pass (0 failed)
• Coverage ≥ 70% enforced
• Cython extensions build cleanly
• HTML reports render correctly in Chrome, Firefox, Safari
• Colorblind theme verified (WCAG AA contrast)
• All four file formats supported (CSV, TSV, JSON, Parquet)
• Docker image builds successfully
• results.json written on every successful run
• No breaking changes to CLI interface
• README and CODE_OF_CONDUCT up to date
• 18 open issues tracked for v1.1 (see above)