Skip to content

Enable predictions on dataset partitions and remove refitting on validation for forecasting - #858

Open
Irozuku wants to merge 6 commits into
developfrom
feat/predict-on-run-splits
Open

Enable predictions on dataset partitions and remove refitting on validation for forecasting#858
Irozuku wants to merge 6 commits into
developfrom
feat/predict-on-run-splits

Conversation

@Irozuku

@Irozuku Irozuku commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

A prediction can now target a single partition of the dataset a run was trained on, instead of always covering every row.

For forecasting that is what makes the results table checkable. Forecasters refuse dates inside their own fit, so a run's own partitions were the interesting ones to ask about and there was no way to ask. Predicting the validation or test rows now returns the same numbers the run recorded, because both go through the model fitted on the training partition and nothing else.

The forecasting holdout strategy no longer refits through validation before scoring test. That refit produced better test metrics by keeping a model nobody could reproduce: the validation metrics came from a fit the refit then overwrote.


Type of Change

Check all that apply like this [x]:

  • Backend change
  • Frontend change
  • CI / Workflow change
  • Build / Packaging change
  • Bug fix
  • Documentation

Changes (by file)

Forecasting models

  • DashAI/back/models/forecasting/base_forecasting_model.py: predict moves to the base class and subclasses supply _forecast(steps), rather than each repeating the same four lines. _dates_of and _steps_of are split out of _steps_ahead. _extend goes with the refit that used it.
  • DashAI/back/models/forecasting/{naive,seasonal_naive,arima,exponential_smoothing}.py: each keeps only what it does differently, which is how it forecasts a number of steps.

Evaluation

  • DashAI/back/evaluation/forecasting_holdout.py: the refit through validation is gone. The kept model is fitted on the training partition alone, like every other holdout run, so the two metric columns describe different horizons and the docstring says so outright.
  • DashAI/back/evaluation/base_evaluation_strategy.py: strategies declare FINAL_FIT_PARTITIONS, the partitions the saved model was fitted through.
  • DashAI/back/evaluation/holdout.py: comment trimmed.

Which partitions a run can be asked about

  • DashAI/back/splitters/splits_payload.py: run_splits names the partitions of a run, run_split_indexes resolves one to row indexes, and predictable_splits narrows that list for a task whose models only predict forward, keeping the partitions that start after the last row the saved model was fitted through. Payloads are accepted as dicts or their JSON encoding, and a task or strategy from an uninstalled plugin resolves to None rather than raising.
  • DashAI/back/tasks/base_task.py, DashAI/back/tasks/forecasting_task.py: PREDICTS_FORWARD_ONLY.
  • DashAI/back/api/api_v1/endpoints/explainers.py: reads run_splits instead of resolving the splitter itself, so explaining and predicting agree on what a run's partitions are.

Prediction on a partition

  • DashAI/back/dependencies/database/models.py, DashAI/alembic/versions/a5f2c71e9d40_add_split_to_prediction.py: adds Prediction.split, nullable. Predictions made before the column existed covered the whole dataset, which is what null means, so no backfill is needed.
  • DashAI/back/api/api_v1/schemas/prediction_params.py: split on the creation params.
  • DashAI/back/api/api_v1/endpoints/predict.py: GET /predict/splits/{run_id} returns the partitions and the id of the training dataset, so the frontend knows when they apply.
  • DashAI/back/job/predict_job.py: selects the partition's rows before predicting. Two HTTPException raises inside the job became JobError, which is what a job raises, and the TypeError path now marks the prediction as failed like every other error path.

Frontend

  • DashAI/front/src/api/predict.ts: getPredictionSplits, and split on createPrediction.
  • DashAI/front/src/components/predictions/DatasetSelector.jsx: adds a partition selector, shown only when the chosen dataset is the one the run was trained on and the run offers partitions. Its caption explains why the list is short when a forecasting run offers no whole dataset option, and it reports how many rows the choice covers.
  • DashAI/front/src/components/predictions/PredictionModal.jsx, DashAI/front/src/components/models/DatasetPredictionPanel.jsx: carry the choice through, sending null for the whole dataset.
  • DashAI/front/src/utils/i18n/locales/{en,es,pt,de,zh}/prediction.json: three labels and a pluralised row count.

Hyperparameter search

  • DashAI/back/optimizers/optuna_optimizer.py: a trial that draws a combination the model cannot be fitted with is recorded as failed and the search moves on, rather than taking the whole run down. Only ValueError and ArithmeticError are caught, so a TypeError from a wrong call still stops the run. A search where every trial fails raises with the last failure quoted; one that lost only some trials logs a warning.

Testing (optional)

Forecasting

  1. Train a forecasting run with Temporal Holdout at 0.6 / 0.2 / 0.2. Note the validation and test metrics.
  2. Open the prediction dialog and choose the dataset the run was trained on. A partition selector should appear offering validation and test, with row counts, and no whole dataset option. The caption should say the model forecasts forward only.
  3. Predict the validation partition. Computing the metric over those predictions should give the validation figure from the results table. This is the claim the branch rests on: before it, this prediction was refused outright.
  4. Predict the test partition and check it the same way against the test figure. Both must agree, since the run and the prediction now use the same model fitted on the same rows.
  5. Choose any other dataset. The selector should disappear and the whole file should be predicted.

Everything else

  1. On a regression or classification run, the same selector should offer train, validation, test and the whole dataset, and each should predict the row count it advertises.
  2. Explanations on that run should still list the same partitions. That endpoint now shares its resolution with predictions, so a mismatch between the two lists is the thing to look for.

Hyperparameter search

  1. Run an optimizer over ARIMA with a wide enough order range that some combinations cannot be fitted. The search should finish, log how many trials it lost, and return a model.
  2. Narrow the range so nothing fits, for example an order far larger than the series is long. It should fail with a message naming the last failure, not a traceback from inside optuna.

Migration

  1. Open a prediction made before this branch. Its split is null, which means the whole dataset, and it should display as it always did.

@Irozuku Irozuku added bug Something isn't working front Frontend work back Backend work labels Sep 2, 2026
@Irozuku
Irozuku marked this pull request as ready for review September 2, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

back Backend work bug Something isn't working front Frontend work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant