csd_check

Validate a critical slowing down / early-warning pipeline on synthetic controls before trusting it on real data.

This package exists because an analysis in this project initially reported a highly significant precursor (p = 1e-5) on a series that contained no precursor at all. Three separate defects in the pipeline caused that, and only a synthetic positive/negative control exposed them:

defect	symptom	fix in this package
90%-overlapping windows fed to a rank test	independence violated; noise looks significant	windowed_stats defaults to disjoint windows
least-squares fit through log of only positive ACF values	white noise scored an autocorrelation time of ~21 samples	autocorrelation_time uses Geyer's initial positive sequence
try/except around a library call	a wrong keyword argument silently looked like "no changepoint found"	errors propagate and surface in check_pipeline()["problems"]

Install

pip install csd_check/            # or: pip install -e csd_check/  for development

Requires Python >= 3.9, numpy, scipy. No other dependencies.

Quick start

from csd_check import check_pipeline, default_pipeline, generate_synthetic_csd

report = check_pipeline(default_pipeline, replicates=20)
print(report["verdict"])                 # OK | MARGINAL ... | UNSPECIFIC ... | INSENSITIVE ...
print(report["false_positive_rate"])     # rate on noise / step controls (want ~alpha)
print(report["detection_rate"])          # rate on the planted-CSD control (want ~1)

Score your own pipeline by passing any callable with this signature:

from csd_check import check_pipeline

def my_pipeline(steps, values, jump_step):
    ...
    return {"p_rise": 0.003}      # one-sided p for "the statistic is rising"

print(check_pipeline(my_pipeline, replicates=20))

check_pipeline accepts a bare float instead of a dict, so an existing function usually needs no adaptation. Anything it raises is reported rather than swallowed.

What is in the box

object	purpose
generate_synthetic_csd(kind, n_points, ...)	known-answer signals: "noise", "step" (no precursor), "ramp", "csd" (planted critical slowing down). Returns steps, values, jump_step and a truth dict
autocorrelation_time(x)	integrated autocorrelation time in samples (Geyer)
windowed_stats(steps, values, window, stride=None)	per-window mean/variance/tau; disjoint by default
trend_test(y)	Spearman rho plus a one-sided "rising" p-value — the right instrument when the statistic has a strong global drift
default_pipeline	a minimal, defensible reference pipeline to compare against
check_pipeline(fn, replicates=...)	false-positive rate, detection rate, Clopper-Pearson CIs, a verdict, and any crashes

Tests and example

python tests/test_csd_check.py          # 11 tests, no pytest required
python -m pytest tests -q               # same tests under pytest, if installed
python examples/usage.py                # good pipeline vs the flawed one, side by side

Measured behaviour of the shipped reference pipeline on this machine:

type-I error over 200 replicates of each negative control: 0.040 (noise), 0.040 (step)
check_pipeline(default_pipeline, replicates=40, n_points=500):
   verdict=OK   false-positive rate=0.025   detection rate=1.000  (40/40 on the CSD control)
check_pipeline(<the flawed overlapping-window variant>, same settings):
   verdict=UNSPECIFIC (fires on pure noise)  false-positive rate=0.45

The 0.45 figure is not hypothetical: it is what the pipeline used early in this project did to a control containing no precursor. An earlier revision of default_pipeline also failed its own calibration (measured type-I error 0.10 rather than 0.05) because it took the minimum of two correlated one-sided p-values without correcting for the two tests; that correction is now in check.default_pipeline.

Interpreting the verdict

verdict	meaning	what to do
OK	fires rarely on controls, reliably on the planted signal	proceed, but report the rates
UNSPECIFIC	fires on noise/step controls	fix the test's independence or the estimator's bias
INSENSITIVE	misses the planted signal	the effect size or window length is wrong for your data
MARGINAL	between the two	raise replicates; rates are binomially noisy below ~20

A caution printed in every report object: use >= 20 replicates before quoting these rates in a paper. At 6 replicates the 95% CI on a 0/6 false-positive count already reaches 0.46.

Scope

csd_check validates the pipeline, not the phenomenon. Passing every control here does not establish that a real transition is critical; it only establishes that your detector is not manufacturing the signal. Conversely, a pipeline that fails here cannot support a real-data claim.

MIT licensed.