Add a short report with performance degradations

[ligurio]

Apache Otava allows analyzing performance results and identify degradations. We use otava analyze command for every test with a period in 30 days and Otava prints a huge list of commits that should be reviewed by someone to identify degradations. Like below, but our list is huge (see https://github.com/tarantool/tarantool/actions/runs/22661725943/job/65683369918):

$ otava analyze local.sample --since=2024-01-01
INFO: Computing change points for test sample.csv...
sample:
time                         metric1    metric2
-------------------------  ---------  ---------
2021-01-01 02:00:00 +0000     154023      10.43
2021-01-02 02:00:00 +0000     138455      10.23
2021-01-03 02:00:00 +0000     143112      10.29
2021-01-04 02:00:00 +0000     149190      10.91
2021-01-05 02:00:00 +0000     132098      10.34
2021-01-06 02:00:00 +0000     151344      10.69
                                      ·········
                                         -12.9%
                                      ·········
2021-01-07 02:00:00 +0000     155145       9.23
2021-01-08 02:00:00 +0000     148889       9.11
2021-01-09 02:00:00 +0000     149466       9.13
2021-01-10 02:00:00 +0000     148209       9.03

This work is time-consuming and routine. Is there a way to print quite and short report with degradations and return non-zero exit code to trigger a fail in CI when degradations detected?

Maybe I missed something in the documentation and what I need has already been implemented?

[Gerrrr]

There are --output regressions_only and --output json. Here is an example. While, the default analysis looks something like this:

time                       commit      metric1    metric2
-------------------------  --------  ---------  ---------
2026-01-01 02:00:00 +0000  aaa0         154023      10.43
2026-01-02 02:00:00 +0000  aaa1         138455      10.23
2026-01-03 02:00:00 +0000  aaa2         143112      10.29
2026-01-04 02:00:00 +0000  aaa3         149190      10.91
2026-01-05 02:00:00 +0000  aaa4         132098      10.34
2026-01-06 02:00:00 +0000  aaa5         151344      10.69
                                                ·········
                                                   -12.9%
                                                ·········
2026-01-07 02:00:00 +0000  aaa6         155145       9.23
2026-01-08 02:00:00 +0000  aaa7         148889       9.11
2026-01-09 02:00:00 +0000  aaa8         149466       9.13
2026-01-10 02:00:00 +0000  aaa9         148209       9.03

With --output regressions_only it will look like:

Computing change points for test local.sample...
Regressions in local.sample:
2026-01-07 02:00:00 +0000
    metric2         :       10.5        -->     9.12    ( -12.9%)

If there are no regressions, it will print:

Computing change points for test local.sample...
No regressions found in local.sample.

You can do string matching in the CI script and exit 1 if there are regressions. For example, here is what we have in one of the pipelines:

if grep "Regressions in" otava.out; then
  echo "Found performance regressions on branch ${BRANCH}. <Link to runbook>"
  exit 1;
fi

With --output json:

Computing change points for test local.sample...
{"local.sample": [{"time": 1767751200, "attributes": {"commit": "aaa6"}, "changes": [{"metric": "metric2", "index": 6, "time": 1767751200, "forward_change_percent": "-13", "magnitude": "0.148676", "mean_before": "10.481667", "stddev_before": "0.241414", "mean_after": "9.125000", "stddev_after": "0.071239", "pvalue": "0.000005"}]}]}

If there are no regressions:

Computing change points for test local.sample...
{"local.sample": []}

If you have jq available in the CI script, you can check that values are empty lists.

I hope one of these options works for you!

Admittedly, I added --output regressions_only to my fork a long time ago without giving much thought to the API. As the project is now moving toward a 1.0.0 release with a stable API, this is the best time to reconsider the ergonomics of this and other options. Let's use this issue to discuss what the ideal CLI args would look like.

One problem with the current --output option is that it ties the output format (tabular, JSON, etc.) to filtering. For example, it might be useful to also have the regular tabular format showing only regressions.

time                       commit      metric1    metric2
-------------------------  --------  ---------  ---------
2026-01-06 02:00:00 +0000  aaa5         151344      10.69
                                                ·········
                                                   -12.9%
                                                ·········
2026-01-07 02:00:00 +0000  aaa6         155145       9.23

Putting it another way, there is no way to filter for the specific information of interest - the entire history, only regressions, only improvements, or all change points.

Maybe, we should have 2 distinct options, one to specify the output format, the second is what events we want to see in that output.

[ligurio]

@Gerrrr Alex, many thanks for explanation and sharing your experience with Otava!

[henrikingo]

@ligurio Thanks for using otava and being active here. Getting more input is going to be crucial in the next 6 - 12 months when for the first time ever we work on a common upstream and make some UX changes and improvements. If you're not already, I invite you to join the mailing list where we occasionally have such conversations too...

A couple of comments on your workflow:

In every place I've seen Otava used (and used myself), it was used within some larger web based UI. Unfortunately none of these were open sourced, now are they even publicly available for you to view, although maybe the Datastax way of doing this is to a large extent available in Otava. Namely, in Datastax, benchmark results would be submitted to Prometheus and the associated Grafana dashboard. Otava then has a functionality to read data from Prometheus, compute change points, and write them back into Prometheus as Grafana annotations. (Over time, other databases have been added.)

In MongoDB we had performance graphs directly in the CI system (very similar to what you might have in Jenkins, for example) and also Jira integration so that we could create a Jira issue directly from the change point alert in CI. Further, this included a feature using custom Jira fields, so that when a regression was eventually fixed, the commit sha of the fix would be added to the same jira, so that the regression and the fix were paired together. The CI graphs would get this information in Jira, and fixed change points could be presented with a less urgent color than the ones not addressed. Same for processing false positives. Was another click of a button.

But speaking more generally, I always advocate to store your test results and compute change points outside and after the actual test run. There are several reasons:

• First of all, Otava isn't designed to find a change point immediately after the test. In many cases it might flag a change only after 4-5 more tests were run. It is only at that point you can be certain that the change really was persistant and not some random up or down.
• As always with "big data", you may want to rerun the analytics part on the same data that you already have. For example you may want to change the p-value or some other parameter. Or fix a bug... You want to be able to do this without having to rerun the actual benchmarks.
• If you analyze the 30 most recent points inside your workflow, and there is an actual regression, now you will be alerted of the same regression 30 times, no? If you are saying you only alert / fail the job if it is the most recent point that is the change point, then re-read the first point.
• And if you have two change points within the 30 day window, how would you notice the second one if the job is already failing because of the first change point?

Like I said, unfortunately I'm not aware of any of the Otava based dashboards to be publicly available. Nyrkiö is a commercial SaaS offering providing this same type of graphing plus integration with github pull requests and issues. (I'm unsure about the etiquette here, but it seemed on topic to mention it in this case.) Here is a random example of a pull request comment from Nyrkiö about one benchmark result being significantly slower than before. nyrkio/nyrkio#968 (comment) and here is the same for a push event, in that case an issue is created unodb-dev/unodb#832 The link in the issue is broken, it tried to link to https://nyrkio.com/public/https%3A%2F%2Fgithub.com%2Funodb-dev%2Funodb/master/UnoDB_Benchmarks__x64_?commit=d2ab269909c64723eb5930201bde3bd8b7cefad3&timestamp=1766116904#full_n4_sequential_insert%3Cunodb::benchmark::olc_db%3E/32768

So with this kind of graphing, ability to rerun the analytics for example after fine tuning parameters, and integration with a ticket system, triaging results should be much more pleasant. You should get more than 50% correct alerts, and you should only get them once, and you should be able to mark a change as closed either because it was fixed or considered invalid. (Nyrkiö doesn't do this last bit.)