BaseModel.calculate_metrics skips metrics that return a non-finite value, with a warning that names the usual cause:
# DashAI/back/models/base_model.py:305-312
score = metric.score(y_transformed, y_pred)
if not math.isfinite(score):
logger.warning("Metric %s returned a non-finite value (%s) for split %s "
"(e.g. only one class present in the split). Skipping.", ...)
continue
compute_metrics (line 339) copies that scoring loop without the guard:
# DashAI/back/models/base_model.py:392-393
score = metric.score(y_transformed, y_pred)
results[metric.__name__] = score
It also keeps the same early returns as calculate_metrics — no metrics configured, or no data for the split — returning {} at lines 372, 377 and 383.
CrossValidationEvaluationStrategy.evaluate is the caller, once per fold:
# DashAI/back/evaluation/cv.py:203-206
train_scores = model.compute_metrics(split=SplitEnum.TRAIN)
validation_scores = model.compute_metrics(split=SplitEnum.VALIDATION)
folds_results.append(validation_scores[metric.__name__])
Two consequences:
-
A NaN from one fold poisons the whole trial. folds_results feeds np.mean(folds_results) at cv.py:244, which is the objective value the optimizer sees. One fold containing a single class makes the trial's score NaN, indistinguishable from a genuinely bad trial. calculate_metrics was written to avoid exactly this.
-
An empty dict raises KeyError. If compute_metrics takes any of its three early returns, validation_scores[metric.__name__] fails and the fold loop dies mid-trial, with a traceback pointing at a subscript rather than at the cause.
Suggested fix: share the scoring loop between the two methods, or add the same isfinite skip to compute_metrics. Either way it is worth deciding explicitly what a fold without the goal metric should do — skip that fold, or fail with a message that names the reason — rather than letting a KeyError decide.
Checked against develop @ b3b7296. Happy to send a PR if you tell me which of the two behaviours you want for the missing-metric case.
BaseModel.calculate_metricsskips metrics that return a non-finite value, with a warning that names the usual cause:compute_metrics(line 339) copies that scoring loop without the guard:It also keeps the same early returns as
calculate_metrics— no metrics configured, or no data for the split — returning{}at lines 372, 377 and 383.CrossValidationEvaluationStrategy.evaluateis the caller, once per fold:Two consequences:
A NaN from one fold poisons the whole trial.
folds_resultsfeedsnp.mean(folds_results)atcv.py:244, which is the objective value the optimizer sees. One fold containing a single class makes the trial's score NaN, indistinguishable from a genuinely bad trial.calculate_metricswas written to avoid exactly this.An empty dict raises
KeyError. Ifcompute_metricstakes any of its three early returns,validation_scores[metric.__name__]fails and the fold loop dies mid-trial, with a traceback pointing at a subscript rather than at the cause.Suggested fix: share the scoring loop between the two methods, or add the same
isfiniteskip tocompute_metrics. Either way it is worth deciding explicitly what a fold without the goal metric should do — skip that fold, or fail with a message that names the reason — rather than letting aKeyErrordecide.Checked against
develop@b3b7296. Happy to send a PR if you tell me which of the two behaviours you want for the missing-metric case.