Skip to content

Add accuracy tracking to PRs. - #224

Merged
jenskeiner merged 1 commit into
developfrom
feature/accuracy-checks
Jul 1, 2026
Merged

Add accuracy tracking to PRs.#224
jenskeiner merged 1 commit into
developfrom
feature/accuracy-checks

Conversation

@jenskeiner

@jenskeiner jenskeiner commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

CodSpeed benchmarks that run on PRs and compare to the baseline on develop already give us an idea if code changes had any impact on performance. The same is not true for accuracy.

The existing automated tests compute various transformations and compare the results to a reference. If the error exceeds a defined bound, the case fails, and thus ultimately the whole CI run. This way, some inadvertent breaks can be captured.

However, it would be more useful to track changes to accuracy figures more precisely. This could flag degrading or improving accuracy that does not cross a threshold. This is particularly interesting in light of upcoming performance and accuracy work, as in #222.

This PR adds some custom code to add the following:

  • Each in-repo PR gets an automated comment that mentions unchanged/improved/regressed tests in terms of accuracy.
  • The comment also includes a link to a webpage (served via GH Pages) that includes heatmaps with absolute results (aggregated over cases with varying parameters N and M that should not be driving the error as it is calculated) and relative results vs. develop.
  • For PRs only get the comment with the summary. Nothing gets uploaded to GH Pages since Fork PRs run with read-only privileges.

At this stage, this new way of presenting accuracy figures seems to be working ok, although I would imagine there's still a lot of room for improvement. The goal of getting this in now is to gather more evidence that the changes in #222 not only improve performance of the direct 1D NDFT transforms, but also the accuracy by avoiding the accumulation of round-off errors.

Comment thread tests/bench/report.py Fixed
Comment thread tests/bench/test_diff.py Fixed
@github-actions

Copy link
Copy Markdown

Accuracy report

No develop baseline yet to compare against — showing this PR's absolute accuracy in the full report. A baseline appears once changes land on develop.

📊 Full accuracy report

@codspeed-hq

codspeed-hq Bot commented Jun 30, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 132 untouched benchmarks


Comparing feature/accuracy-checks (cfdc22b) with develop (56ea7dc)

Open in CodSpeed

@jenskeiner
jenskeiner force-pushed the feature/accuracy-checks branch from 77e3a1e to cfdc22b Compare July 1, 2026 08:39
@jenskeiner jenskeiner changed the title Feature/accuracy checks Add accuracy tracking to PRs. Jul 1, 2026
@jenskeiner
jenskeiner marked this pull request as ready for review July 1, 2026 10:03
@jenskeiner
jenskeiner merged commit a6bb4bb into develop Jul 1, 2026
39 checks passed
@jenskeiner
jenskeiner deleted the feature/accuracy-checks branch July 3, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant