Skip to content

Compute and output normalized error above tolerance - #112

Open
stelliom wants to merge 1 commit into
mainfrom
normalized-err
Open

Compute and output normalized error above tolerance#112
stelliom wants to merge 1 commit into
mainfrom
normalized-err

Conversation

@stelliom

Copy link
Copy Markdown
Collaborator

This PR adds the computation of a fourth metric for the output of check.py. Instead of only showing the "Error relative to tolerance":

err_rel_tol = err / tol

We also show the "Normalized error above tolerance":

err_above_tol = (err - tol) / tol = err / tol - 1

This let's user see how much above the tolerance the checked variables are, in a normalized fashion, such that errors for different variables can be compared. This is especially useful when err_rel_tol is very close to 1, since the precision of the output only shows 2 decimal digits. With this new output you can see how much above the tolerance the values are even if they are only slightly above (i.e. err_rel_tol ~ 1).

Note that values that do not exceed the tolerance are clipped to 0, this allows for spotting errors quicker.

@stelliom

Copy link
Copy Markdown
Collaborator Author

@huppd Follow up on our Slack conversation. What do you think?

@huppd huppd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this proposition. I like it :) also the clipping is a good idea.

Comment thread engine/check.py
err_above_tol = (err_rel_tol - 1.).clip(lower=0.)
log_dataframe(
logger,
"\nError relative to tolerance",

@huppd huppd Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"\nError relative to tolerance",
"\nErrors above tolerance (normalized): max((Differences - Tolerances) / Tolerances, 0)",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants