Define the framework for implementing reusable and extensible data quality analysis and rejection pipelines in wf-psf.
The pipeline should support multiple independent data quality metrics and rejection policies while maintaining a single, consistent mechanism for filtering aligned WaveDiff datasets during data preparation.
A key design principle is to decouple metric computation from rejection policy, allowing quality metrics to be reused for diagnostics, validation, and monitoring independently of threshold-based filtering.
quality metric
↓
metric value
↓
rejection policy
↓
boolean mask
Multiple rejection masks are combined into a single validity mask, which is applied to the aligned dataset.
Configuration
|
v
Quality Metric Computation
|
+--> mask obscuration
|
+--> goodness-of-fit
|
v
Quality Metrics
|
v
Rejection Policies
|
v
Combined validity mask
|
v
Dataset filtering
The framework should distinguish between:
- Data validity checks required to construct a usable dataset (e.g. finite-value checks during data preparation).
- Scientific quality criteria used to assess or reject samples based on configurable policies
Goals
Define the framework for implementing reusable and extensible data quality analysis and rejection pipelines in
wf-psf.The pipeline should support multiple independent data quality metrics and rejection policies while maintaining a single, consistent mechanism for filtering aligned WaveDiff datasets during data preparation.
A key design principle is to decouple metric computation from rejection policy, allowing quality metrics to be reused for diagnostics, validation, and monitoring independently of threshold-based filtering.
Multiple rejection masks are combined into a single validity mask, which is applied to the aligned dataset.
The framework should distinguish between:
Goals