Enable predictions on dataset partitions and remove refitting on validation for forecasting - #858
Open
Irozuku wants to merge 6 commits into
Open
Enable predictions on dataset partitions and remove refitting on validation for forecasting#858Irozuku wants to merge 6 commits into
Irozuku wants to merge 6 commits into
Conversation
Irozuku
marked this pull request as ready for review
September 2, 2026 21:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A prediction can now target a single partition of the dataset a run was trained on, instead of always covering every row.
For forecasting that is what makes the results table checkable. Forecasters refuse dates inside their own fit, so a run's own partitions were the interesting ones to ask about and there was no way to ask. Predicting the validation or test rows now returns the same numbers the run recorded, because both go through the model fitted on the training partition and nothing else.
The forecasting holdout strategy no longer refits through validation before scoring test. That refit produced better test metrics by keeping a model nobody could reproduce: the validation metrics came from a fit the refit then overwrote.
Type of Change
Check all that apply like this [x]:
Changes (by file)
Forecasting models
DashAI/back/models/forecasting/base_forecasting_model.py:predictmoves to the base class and subclasses supply_forecast(steps), rather than each repeating the same four lines._dates_ofand_steps_ofare split out of_steps_ahead._extendgoes with the refit that used it.DashAI/back/models/forecasting/{naive,seasonal_naive,arima,exponential_smoothing}.py: each keeps only what it does differently, which is how it forecasts a number of steps.Evaluation
DashAI/back/evaluation/forecasting_holdout.py: the refit through validation is gone. The kept model is fitted on the training partition alone, like every other holdout run, so the two metric columns describe different horizons and the docstring says so outright.DashAI/back/evaluation/base_evaluation_strategy.py: strategies declareFINAL_FIT_PARTITIONS, the partitions the saved model was fitted through.DashAI/back/evaluation/holdout.py: comment trimmed.Which partitions a run can be asked about
DashAI/back/splitters/splits_payload.py:run_splitsnames the partitions of a run,run_split_indexesresolves one to row indexes, andpredictable_splitsnarrows that list for a task whose models only predict forward, keeping the partitions that start after the last row the saved model was fitted through. Payloads are accepted as dicts or their JSON encoding, and a task or strategy from an uninstalled plugin resolves toNonerather than raising.DashAI/back/tasks/base_task.py,DashAI/back/tasks/forecasting_task.py:PREDICTS_FORWARD_ONLY.DashAI/back/api/api_v1/endpoints/explainers.py: readsrun_splitsinstead of resolving the splitter itself, so explaining and predicting agree on what a run's partitions are.Prediction on a partition
DashAI/back/dependencies/database/models.py,DashAI/alembic/versions/a5f2c71e9d40_add_split_to_prediction.py: addsPrediction.split, nullable. Predictions made before the column existed covered the whole dataset, which is what null means, so no backfill is needed.DashAI/back/api/api_v1/schemas/prediction_params.py:spliton the creation params.DashAI/back/api/api_v1/endpoints/predict.py:GET /predict/splits/{run_id}returns the partitions and the id of the training dataset, so the frontend knows when they apply.DashAI/back/job/predict_job.py: selects the partition's rows before predicting. TwoHTTPExceptionraises inside the job becameJobError, which is what a job raises, and theTypeErrorpath now marks the prediction as failed like every other error path.Frontend
DashAI/front/src/api/predict.ts:getPredictionSplits, andsplitoncreatePrediction.DashAI/front/src/components/predictions/DatasetSelector.jsx: adds a partition selector, shown only when the chosen dataset is the one the run was trained on and the run offers partitions. Its caption explains why the list is short when a forecasting run offers no whole dataset option, and it reports how many rows the choice covers.DashAI/front/src/components/predictions/PredictionModal.jsx,DashAI/front/src/components/models/DatasetPredictionPanel.jsx: carry the choice through, sending null for the whole dataset.DashAI/front/src/utils/i18n/locales/{en,es,pt,de,zh}/prediction.json: three labels and a pluralised row count.Hyperparameter search
DashAI/back/optimizers/optuna_optimizer.py: a trial that draws a combination the model cannot be fitted with is recorded as failed and the search moves on, rather than taking the whole run down. OnlyValueErrorandArithmeticErrorare caught, so aTypeErrorfrom a wrong call still stops the run. A search where every trial fails raises with the last failure quoted; one that lost only some trials logs a warning.Testing (optional)
Forecasting
Everything else
Hyperparameter search
Migration