RFC: where should data-level metrics live (complexity, drift, quality)?

[CristobalSantana]

Motivation

In DashAI, the metrics are focused on measuring the result of the models, not the data. There are metrics for the data, and they are usually used before training the models: complexity measures for class overlap (Ho & Basu, 2002; Lorena et al., 2019) and dataset shift for drift (Gama et al., 2014). Today DashAI has no place for this, and which would be the right place is not so clear to me, so I am opening this issue instead of a PR.

Where they could go

Not in the metric hierarchy: BaseMetric requires score(true, pred) -> float and assumes one number per run (DashAI/back/metrics/base_metric.py). A complexity measure has no pred and no run. Adding it there is perhaps not the most appropriate option, given the structure/architecture of the software.

Maybe in the exploration subsystem: BaseExplorer already works on datasets, saves results and shows them (DashAI/back/exploration/). A "Data Complexity" category could be an option.

My doubt: explorers are visualisations that are looked at once, while these values matter above all when they are stored and compared across datasets.

Or a new component type (DataMetrics): a dataset goes in, a scalar or a per-column result comes out.

Scope

• Class overlap: F1, F2, N1, N2, N3 from Lorena et al.
• Imbalance and quality: imbalance ratio, missing values, cardinality, duplicates, constant columns, outliers.
• Leakage and redundancy: unusual association between feature and target, near-duplicate features, contamination between train and test.
• Drift: PSI and KS for numeric columns, Jensen-Shannon or chi-square for categorical ones.

Current considerations

Explorer.notebook_id points to a single Notebook, and Notebook.dataset_id to a single Dataset (DashAI/back/dependencies/database/models.py), so there is no way to compare dataset A with dataset B. Possible solution: keep drift within a single dataset, or add a second dataset id as a schema parameter.

Once this is settled I can write a concrete design. For context, I recently added the BART regression model and the Matthews correlation coefficient metric, so my doubt is architectural. If you give it the go-ahead, I would appreciate knowing what the structure for this will be, so that I can send a PR that follows that structure and start contributing to this new section.

[cristian-tamblay]

Thanks for writing this up as an issue instead of a PR, the analysis is accurate and it saved us a round trip.

We discussed it with the devs. We want to keep it on the roadmap, but we are not in a position to hand you a target structure yet. Where things stand:

You are right that BaseMetric is not the place. score(true, pred) -> float is tied to a run, and metric values are stored per run and read back through the run comparison views. A measure with no prediction and no run would have to fake both to fit there.

The comparison part is the real blocker, and it is more a UI problem than a model one. Today the only comparison surface in dashAI is between models/runs inside a model session. There is nothing that puts two datasets side by side.

Also we are not sure if we can compare two distinct datasets, or they should share at least some of the metadata.

So: we are leaving the issue open and will come back to it once we have a decision on the comparison surface. If you want to move in the meantime, the single-dataset measures are the part least likely to be invalidated by whatever we settle on