Skip to content

compute_metrics keeps non-finite scores that calculate_metrics filters, and the CV objective can crash on an empty result #843

Description

@Maarmapa

BaseModel.calculate_metrics skips metrics that return a non-finite value, with a warning that names the usual cause:

# DashAI/back/models/base_model.py:305-312
score = metric.score(y_transformed, y_pred)
if not math.isfinite(score):
    logger.warning("Metric %s returned a non-finite value (%s) for split %s "
                   "(e.g. only one class present in the split). Skipping.", ...)
    continue

compute_metrics (line 339) copies that scoring loop without the guard:

# DashAI/back/models/base_model.py:392-393
score = metric.score(y_transformed, y_pred)
results[metric.__name__] = score

It also keeps the same early returns as calculate_metrics — no metrics configured, or no data for the split — returning {} at lines 372, 377 and 383.

CrossValidationEvaluationStrategy.evaluate is the caller, once per fold:

# DashAI/back/evaluation/cv.py:203-206
train_scores = model.compute_metrics(split=SplitEnum.TRAIN)
validation_scores = model.compute_metrics(split=SplitEnum.VALIDATION)
folds_results.append(validation_scores[metric.__name__])

Two consequences:

  1. A NaN from one fold poisons the whole trial. folds_results feeds np.mean(folds_results) at cv.py:244, which is the objective value the optimizer sees. One fold containing a single class makes the trial's score NaN, indistinguishable from a genuinely bad trial. calculate_metrics was written to avoid exactly this.

  2. An empty dict raises KeyError. If compute_metrics takes any of its three early returns, validation_scores[metric.__name__] fails and the fold loop dies mid-trial, with a traceback pointing at a subscript rather than at the cause.

Suggested fix: share the scoring loop between the two methods, or add the same isfinite skip to compute_metrics. Either way it is worth deciding explicitly what a fold without the goal metric should do — skip that fold, or fail with a message that names the reason — rather than letting a KeyError decide.

Checked against develop @ b3b7296. Happy to send a PR if you tell me which of the two behaviours you want for the missing-metric case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions