data_drift: A Python Package for Detecting Data Drift

What is data drift?

Data drift occurs when the underlying distribution of your data changes over time. This can happen due to various factors like seasonal variations, changes in user behavior, or updates in data collection methods. Data drift can negatively impact the performance of machine learning models trained on historical data. Models that rely on assumptions about the data distribution may produce inaccurate predictions if the data distribution has shifted significantly.

Introducing data_drift

The data_drift package provides a Python class, DriftDetector, to help you detect and quantify data drift between two pandas DataFrames. It calculates statistical metrics to measure the difference in distributions and helps you identify potential issues in your data.

Key Features:

Supports various metrics: Calculate drift for numerical data using metrics like mean, median, standard deviation, and more. For categorical data, use the mode metric as of now. Granular analysis: Detect drift at different levels of granularity by grouping data based on a specified column (e.g., date, region). This helps you understand how drift patterns might vary across different subsets of your data.

Easy to use: The DriftDetector class simplifies drift detection with intuitive functions.

Installation:

Clone or download the data_drift package.

Install the package and its dependencies using pip (assuming you have a requirements.txt file listing dependencies):

pip install -r requirements.txt

Import the DriftDetector class:

from data_drift import DriftDetector

Create DriftDetector objects with your DataFrames:
old_data = pd.read_csv("data_old.csv")
new_data = pd.read_csv("data_new.csv")
detector = DriftDetector(old_data, new_data)

Specify the metrics you want to calculate drift for:
detector.set_metrics(["mean", "median", "std_dev"])    # For numerical data
detector.set_metrics(["mode"])                         # For categorical data

Calculate drift for specific columns or with granularity:

# Overall drift for columns 'col1' and 'col2'
drift_metrics = detector.calculate_drift(["col1", "col2"])

# Drift by 'date' group for column 'col3'
drift_metrics = detector.calculate_drift(["col3"], granularity="date")

## The calculate_drift method returns a dictionary containing drift metrics for each column, metric, and granularity level.

drift_metrics = detector.calculate_drift(["col1", "col2"])
print(drift_metrics)

This will print a dictionary like:

{
  "col1": {
    "mean_diff": 0.5,
    "median_diff": 0.2,
    "std_dev_diff": 1.0
  },
  "col2": {
    "mode_diff": "category_A"
  }
}

The output shows the difference in mean, median, and standard deviation for col1 and the difference in mode for col2 between the two DataFrames.

Benefits:

Proactive monitoring: Detect data drift early to prevent model degradation. By identifying drift, you can take steps to retrain your models or address the underlying cause of the drift. Improved model performance: Re-train models when drift is significant to maintain accuracy. Models that are not adjusted for data drift may produce inaccurate predictions. Better data understanding: Gain insights into changes in your data over time. Understanding how your data distribution evolves can help you make informed decisions about data collection, model selection, and feature engineering.

Further Exploration:

Consider using the data_drift package as part of a data quality monitoring pipeline. Explore more advanced drift detection techniques beyond statistical metrics.