feat: add gift-eval exp replication

.github/workflows/ci.yaml

@@ -119,6 +119,9 @@ jobs:
         with:
           enable-cache: true
 
+      - name: Install gift-eval deps (editable foundationforecast)
+        run: uv sync --frozen --no-dev
+
       - name: Run GIFT-Eval CI subset on Modal
         run: uv run modal run -m src.runners.run_modal::run_ci
         env:

README.md

@@ -51,6 +51,12 @@ Your DataFrame needs three columns: `unique_id`, `ds`, and `y`. For best results
 
 ---
 
+## Highlights
+
+- 🎯 **Reproducible by design.** FoundationForecast implementations are regression-tested against [official GIFT-Eval submissions](https://huggingface.co/spaces/Salesforce/GIFT-Eval) to ensure they continue to reproduce their benchmark behavior. [CI](https://github.com/TimeCopilot/foundationforecast/actions/workflows/ci.yaml) automatically re-runs [`experiments/gift-eval`](experiments/gift-eval) on Modal GPU and verifies MASE and CRPS against Hugging Face reference CSVs for every change.
+
+---
+
 ## Supported models
 
 Every model supports **forecast**, **cross-validation**, and **anomaly detection** through the same API. **Intervals** means prediction intervals via `level` or quantile forecasts. **Finetuning** marks models that can adapt to your data at inference time. **License** is the [weight/checkpoint license](https://huggingface.co/models) on the default Hugging Face repo (or provider terms for hosted APIs). See the note below for production use.

docs/examples/anomaly-detection-forecaster-quickstart.ipynb

@@ -35,6 +35,7 @@
    "outputs": [],
    "source": [
     "import pandas as pd\n",
+    "\n",
     "from foundationforecast import FoundationForecast"
    ]
   },

docs/examples/finetuning.ipynb

@@ -37,16 +37,15 @@
    },
    "outputs": [],
    "source": [
-    "import os\n",
     "\n",
-    "import pandas as pd\n",
     "from functools import partial\n",
     "\n",
-    "from foundationforecast import FoundationForecast\n",
-    "from foundationforecast.models.chronos import Chronos, ChronosFinetuningConfig\n",
-    "\n",
+    "import pandas as pd\n",
     "from utilsforecast.evaluation import evaluate\n",
-    "from utilsforecast.losses import mase, mape, scaled_crps\n"
+    "from utilsforecast.losses import mape, mase, scaled_crps\n",
+    "\n",
+    "from foundationforecast import FoundationForecast\n",
+    "from foundationforecast.models.chronos import Chronos, ChronosFinetuningConfig\n"
    ]
   },
   {
@@ -98,9 +97,9 @@
     "| Parameter | Type | Default | Description |\n",
     "|-----------|------|---------|-------------|\n",
     "| `finetune_steps` | int | 1000 | Number of training steps. Maps to the chronos pipeline's `num_steps`. |\n",
-    "| `learning_rate` | float or None | None \u2192 1e-6 | Optimizer learning rate (chronos uses 1e-6; for LoRA, 1e-5 is recommended). |\n",
-    "| `batch_size` | int or None | None \u2192 256 | Training batch size for finetuning. The `batch_size` on `Chronos` is for inference only. |\n",
-    "| `finetune_mode` | \"full\" or \"lora\" or None | None \u2192 \"full\" | Full parameter update vs. LoRA. |\n",
+    "| `learning_rate` | float or None | None → 1e-6 | Optimizer learning rate (chronos uses 1e-6; for LoRA, 1e-5 is recommended). |\n",
+    "| `batch_size` | int or None | None → 256 | Training batch size for finetuning. The `batch_size` on `Chronos` is for inference only. |\n",
+    "| `finetune_mode` | \"full\" or \"lora\" or None | None → \"full\" | Full parameter update vs. LoRA. |\n",
     "| `lora_config` | object or None | None | LoRA configuration when `finetune_mode=\"lora\"`; see the [Chronos-2 quickstart](https://github.com/amazon-science/chronos-forecasting/blob/main/notebooks/chronos-2-quickstart.ipynb) for details. |\n",
     "| `save_path` | str or Path or None | None | If set, the finetuned model is saved to this directory. Use the same path as `repo_id` with `finetuning_config=None` to load and reuse it for later forecasts. |\n",
     "\n",
@@ -380,7 +379,7 @@
     "h = 24\n",
     "n_windows = 4\n",
     "\n",
-    "# Baseline (no finetuning) + 4 finetune step values \u2014 one forecaster per config\n",
+    "# Baseline (no finetuning) + 4 finetune step values — one forecaster per config\n",
     "finetune_steps_list = range(10, 50, 10)\n",
     "models_eval = [\n",
     "    Chronos(repo_id=\"autogluon/chronos-2-small\", alias=\"Chronos2-baseline\"),\n",
@@ -488,7 +487,7 @@
     "|-----------|------|---------|-------------|\n",
     "| `finetune_steps` | int | 10 | Number of training iterations to minimize forecasting error. |\n",
     "| `finetune_loss` | `\"default\"`, `\"mae\"`, `\"mse\"`, `\"rmse\"`, `\"mape\"`, or `\"smape\"` | `\"default\"` | Loss function used during finetuning. |\n",
-    "| `finetune_depth` | 1\u20135 | 1 | How many model layers to finetune (1 = few, 5 = entire model). |"
+    "| `finetune_depth` | 1–5 | 1 | How many model layers to finetune (1 = few, 5 = entire model). |"
    ]
   },
   {
@@ -587,9 +586,9 @@
        "description": "",
        "description_tooltip": null,
        "layout": "IPY_MODEL_83cee7deed884b5caa6ca3d0284d8bae",
-       "placeholder": "\u200b",
+       "placeholder": "​",
        "style": "IPY_MODEL_d789762af87c4ff1b8c3dca72822eeb5",
-       "value": "config.json:\u2007100%"
+       "value": "config.json: 100%"
       }
      },
      "1a7b931fe2694c8cac081035894ea0fe": {
@@ -890,9 +889,9 @@
        "description": "",
        "description_tooltip": null,
        "layout": "IPY_MODEL_260439f5cc69483bab3b43dad0694699",
-       "placeholder": "\u200b",
+       "placeholder": "​",
        "style": "IPY_MODEL_e1cabf3c599440c3b52c2f6c14324659",
-       "value": "model.safetensors:\u2007100%"
+       "value": "model.safetensors: 100%"
       }
      },
      "30b8d77abf5b4050bacdb12c2b6f4117": {
@@ -926,9 +925,9 @@
        "description": "",
        "description_tooltip": null,
        "layout": "IPY_MODEL_2c821b0006e94b0aa0aabcc1e9ee0d08",
-       "placeholder": "\u200b",
+       "placeholder": "​",
        "style": "IPY_MODEL_30b8d77abf5b4050bacdb12c2b6f4117",
-       "value": "\u2007969/969\u2007[00:00<00:00,\u2007105kB/s]"
+       "value": " 969/969 [00:00<00:00, 105kB/s]"
       }
      },
      "65f5c04a6d5d4c4d8306b0d615bc2e62": {
@@ -1153,9 +1152,9 @@
        "description": "",
        "description_tooltip": null,
        "layout": "IPY_MODEL_1e04f1bedfa8462ea2c8d3ddceb8b553",
-       "placeholder": "\u200b",
+       "placeholder": "​",
        "style": "IPY_MODEL_dbca7312f2c841748a6ca1db5aa6714a",
-       "value": "\u2007112M/112M\u2007[00:01<00:00,\u2007169MB/s]"
+       "value": " 112M/112M [00:01<00:00, 169MB/s]"
       }
      },
      "d0ef379b2d664c578e5fed15354ae6e6": {

docs/examples/forecaster-quickstart.ipynb

@@ -34,6 +34,7 @@
    ],
    "source": [
     "import pandas as pd\n",
+    "\n",
     "from foundationforecast import FoundationForecast"
    ]
   },

experiments/gift-eval/Makefile

@@ -1,4 +1,4 @@
-.PHONY: download-gift-eval-data upload-data-to-s3 sync-ci-results
+.PHONY: download-gift-eval-data upload-data-to-s3 sync-ci-results sync-results verify-all replication-table
 
 download-gift-eval-data:
 	@hf download Salesforce/GiftEval --repo-type=dataset --local-dir=./data/gift-eval
@@ -9,3 +9,14 @@ upload-data-to-s3: download-gift-eval-data
 sync-ci-results:
 	@mkdir -p ./results/ci
 	@aws s3 sync s3://foundationforecast-gift-eval/results/ci ./results/ci
+
+sync-results:
+	@mkdir -p ./results
+	@aws s3 sync s3://foundationforecast-gift-eval/results ./results
+
+verify-all: sync-results
+	@uv run python -m src.runners.run_verify --all
+
+replication-table: sync-results
+	@uv run python -m src.runners.run_verify --all --verify-only \
+		--table-output ./results/replication_table.csv

experiments/gift-eval/README.md

@@ -20,6 +20,8 @@ cd experiments/gift-eval
 uv sync
 ```
 
+Installs the in-repo editable `foundationforecast` package from the monorepo root (`../..`), not PyPI — so local runs and CI always use the current wrapper code.
+
 Requires Python 3.11+.
 
 ## Dataset
@@ -33,24 +35,33 @@ make download-gift-eval-data
 
 ```bash
 uv run python -m src.runners.run_model \
-  --model-key chronos-bolt-small \
+  --model-key amazon--chronos-bolt-small \
   --dataset-name m4_weekly \
   --term short \
   --storage-path ./data/gift-eval \
   --output-root ./results
 ```
 
-## CI subset (local GPU)
+## CI subset
+
+[`configs/ci_subset.yaml`](configs/ci_subset.yaml) defines **10 jobs**: Chronos on
+`m4_weekly/short` and `m4_hourly/short`, plus one representative `model_key` each for
+TimesFM, TiRex, Moirai, Toto, FlowState, PatchTST-FM, T0, and Tafsut (all on `m4_weekly/short`).
+Each job runs on Modal GPU and is **HF-verified** in pytest (metrics must match the
+official GIFT-Eval reference CSV).
+
+### Local GPU
 
 ```bash
 uv run python -m src.runners.run_ci --local --verify \
   --storage-path ./data/gift-eval \
   --output-root ./results/ci
 ```
 
-## CI subset (Modal)
+### Modal (CI / GitHub Actions)
 
-Always re-runs and overwrites results (no skip-if-exists). Full grid skips jobs that already have outputs.
+Always re-runs and overwrites results (no skip-if-exists). Full grid skips jobs that
+already have outputs.
 
 ```bash
 uv run modal run -m src.runners.run_modal::run_ci
@@ -66,21 +77,106 @@ One GPU job per `(model_key, dataset, term)`:
 uv run modal run -m src.runners.run_modal::main
 ```
 
+## Verify against HF references
+
+Compare local/S3 results to official GIFT-Eval CSVs. Uses consolidated
+`results/{model_key}/all_results.csv` if present, otherwise aggregates
+per-job CSVs under `results/{model_key}/`.
+
+Strict replication asserts **MASE** and **CRPS** only (the GIFT-Eval ranking
+metrics), with default tolerances `atol=0.01`, `rtol=0.025`. Other columns in
+`all_results.csv` are still written but not compared.
+
+Every verify run also writes a replication analysis table (CSV) with:
+
+| Column | Description |
+|--------|-------------|
+| `dataset` | GIFT-Eval dataset config (e.g. `m4_weekly/W/short`) |
+| `model` | Model alias in results CSV |
+| `model_key` | Experiment registry key |
+| `time_seconds` | Eval wall time (from per-job `timing.json`) |
+| `mase` | Our `eval_metrics/MASE[0.5]` |
+| `crps` | Our `eval_metrics/mean_weighted_sum_quantile_loss` |
+| `reported_gift_eval_mase` | Official HF reference MASE |
+| `reported_gift_eval_crps` | Official HF reference CRPS |
+| `mase_diff` | `mase - reported_gift_eval_mase` |
+| `crps_diff` | `crps - reported_gift_eval_crps` |
+
+```bash
+# CI subset (per-job layout under results/ci/)
+uv run python -m src.runners.run_verify --ci
+
+# One model
+uv run python -m src.runners.run_verify --model-key amazon--chronos-bolt-small
+
+# All models with a reference_slug in configs/models.yaml
+make sync-results   # or: aws s3 sync s3://foundationforecast-gift-eval/results ./results
+uv run python -m src.runners.run_verify --all
+
+# Table only (no strict assert) — good for exploratory analysis
+uv run python -m src.runners.run_verify --all --verify-only \
+  --table-output ./results/replication_table.csv
+make replication-table
+
+# Require every HF dataset to be present (not just compare overlap)
+uv run python -m src.runners.run_verify --all --require-complete
+```
+
+Or in one step:
+
+```bash
+make verify-all
+```
+
+**Note:** `time_seconds` is recorded when a job runs via `run_gift_eval` (writes
+`timing.json` next to each `all_results.csv`). To backfill timing for jobs that
+ran before timing was added:
+
+```bash
+# Full grid: rerun only jobs with results but no timing.json on S3
+uv run modal run -m src.runners.run_modal::run_missing_timing
+
+# Full grid: force rerun everything (also refreshes metrics)
+uv run modal run -m src.runners.run_modal::main --force
+
+# CI subset locally
+uv run python -m src.runners.run_ci --local --missing-timing-only
+
+# CI on Modal always reruns with force=True (timing included every CI run)
+uv run modal run -m src.runners.run_modal::run_ci
+```
+
+Then sync and rebuild the table:
+
+```bash
+make sync-results
+make replication-table
+```
+
 ## Consolidate S3 results
 
 ```bash
-uv run python -m src.runners.download_results --model-key chronos-bolt-small
+uv run python -m src.runners.download_results --model-key amazon--chronos-bolt-small
 ```
 
 ## Infrastructure
 
 - **S3 bucket:** `foundationforecast-gift-eval`
-- **Modal secret:** `aws-secret` (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
+- **Modal secrets:**
+  - `aws-secret` — `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
+  - `hf-secret` — `HF_TOKEN` (required for gated models like `t0-alpha`; create with
+    `modal secret create hf-secret HF_TOKEN=hf_...`)
 - **Modal tokens:** `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`
-- **Hugging Face:** `HF_TOKEN` (dataset + model weights)
+- **Hugging Face:** accept model licenses on the Hub, then set `HF_TOKEN` in `hf-secret`
 
 ## Adding a model
 
-1. Add an entry to `configs/models.yaml` with `class`, `kwargs`, and `reference_slug`.
-2. Match `alias` to the official GIFT-Eval `model` column in the HF results CSV.
-3. Set `reference_slug: null` if no public reference exists (verify skips that model).
+1. Add an entry to `configs/models.yaml` with slugified `repo_id` as `model_key`
+   (`org--model`), `class`, `kwargs.repo_id`, and `reference_slug`.
+2. Set `kwargs.repo_id` from the official GIFT-Eval
+   `results/{reference_slug}/config.json` → `model_link` (many models use
+   gifteval-specific HF repos, not the default public checkpoint).
+3. Set `reference_slug` to the official GIFT-Eval folder name; `alias` defaults from
+   that in `build_model()` — set `kwargs.alias` explicitly when the CSV `model` column
+   differs from the folder slug (e.g. `chronos_base` → `Chronos_base`).
+4. Set `reference_slug: null` if no public reference exists (verify skips that model).

experiments/gift-eval/configs/ci_subset.yaml

@@ -1,10 +1,33 @@
+# Representative CI subset for replication verification (Chronos on two datasets).
+# Every job is HF-verified in tests/test_replication.py after Modal run_ci.
 jobs:
-  - model_key: chronos-bolt-small
+  - model_key: amazon--chronos-bolt-small
     dataset_name: m4_weekly
     term: short
-  - model_key: chronos-bolt-small
+  - model_key: amazon--chronos-bolt-small
     dataset_name: m4_hourly
     term: short
-  - model_key: timesfm-2.5
+  - model_key: google--timesfm-2.5-200m-pytorch
+    dataset_name: m4_weekly
+    term: short
+  - model_key: NX-AI--TiRex-1.1-gifteval
+    dataset_name: m4_weekly
+    term: short
+  - model_key: Salesforce--moirai-1.1-R-base
+    dataset_name: m4_weekly
+    term: short
+  - model_key: Datadog--Toto-2.0-4m
+    dataset_name: m4_weekly
+    term: short
+  - model_key: ibm-research--flowstate
+    dataset_name: m4_weekly
+    term: short
+  - model_key: ibm-research--patchtst-fm-r1
+    dataset_name: m4_weekly
+    term: short
+  - model_key: theforecastingcompany--t0-alpha
+    dataset_name: m4_weekly
+    term: short
+  - model_key: tafsut-univariate-base
     dataset_name: m4_weekly
     term: short