Restore timeseries benchmark (corrected, consolidated) — point duel + interval study (#57)

[OVVO-Financial]

Summary

Restores the gists/timeseries/conformal benchmark withdrawn under #57, as a single consolidated routine (run_conformal.py). The original (a) leaked each evaluation chunk into NNS.ARMA.optim's validation tail and (b) compared methods across mismatched protocols (online h=1 baselines vs. an NNS multi-step block). Both are fixed: one leak-free block walk-forward emits two coherent analyses from the same NNS forecast. The leak is prevented by correct variable specification at the call site (variable=train_slice, never y[:end_i]).

One protocol

At each origin t, given only data through t, every method forecasts an h-step block (h = implied_h = t·(1−0.9)/0.9) with no online updating — no peeking at a realized value to predict the next one.

NNS's model is selected on a strictly historical validation tail; the scored block is never shown to the optimizer. DGP: heteroskedastic AR(1) with trend, two seasonals, piecewise σ-regimes. 10 seeds, T=3500.

Point duel — does NNS forecast better?

           method   MAE  RMSE  median_AE
        NNS block 1.514 2.056      1.122
Ridge (recursive) 2.664 3.230      2.403
      Persistence 2.493 3.241      1.991

NNS wins decisively (~43% lower MAE than recursive ridge, all 10 seeds). Recursive ridge falls below persistence once the h=1 true-lag crutch is removed — its error compounds over the block.

Interval study — native PI vs. conformalizing the same residuals

Point forecast held fixed; only the band varies. Framed explicitly as an adaptation to discern coverage guarantees on a heteroskedastic process.

                                 method  marg_cov  worst_win_cov  cov_lowvol  cov_hivol  cond_cov_gap  width  interval_score
                          NNS native PI     0.845          0.561       0.927      0.853         0.149  5.658           8.674
                  NNS + split-CP (flat)     0.824          0.515       0.899      0.850         0.169  5.398           8.735
                  NNS + Gaussian (flat)     0.821          0.516       0.894      0.842         0.170  5.332           8.773
              NNS + split-CP (per-lead)     0.918          0.649       0.996      0.858         0.097  8.595          10.505
Ridge (recursive) + split-CP (per-lead)     0.865          0.553       0.955      0.826         0.149 10.169          13.853
      Persistence + split-CP (per-lead)     0.887          0.333       0.981      0.807         0.162 12.107          16.440

• NNS native PI is the efficiency winner and ≈ a flat split-conformal band on the same residuals.
• Every flat band under-covers the volatile regime — exchangeability failure under heteroskedasticity.
• Only the horizon-adaptive per-lead wrapper recovers near-nominal coverage (0.918), at ~52% more width.
• Interval quality follows point quality: same wrapper, NNS 10.5 ≪ ridge 13.9 ≪ persistence 16.4.

Layout

gists/timeseries/conformal/
  run_conformal.py            # one routine -> both tables
  README.md                   # framing + both tables + findings
  results/{point,interval}{,_all}.csv

Note: the interval CRPS/log-score columns were dropped from the consolidated table; lean on interval score + the coverage columns.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TX6vFofX1vnanE147tJFWF