Skip to content

A KPI sums across data sources, so one event reported twice is counted twice #2471

Description

@Flix6x

What happens

A KPI totals one value per source, so when two data sources report the same event, a sum KPI
adds both of them up. The KPI then shows a number that no single source ever reported, and that no
point on the chart shows either.

Example

One sensor, one event (a single day), reported by two sources — say a forecaster's value that was
later corrected by an upload, or two forecasters with different configurations:

event_start source value
2030-03-15 source A 100.0
2030-03-15 source B 80.0

with

asset.sensors_to_show_as_kpis = [
    {"title": "Daily costs", "sensor": sensor.id, "function": "sum"}
]

[GET] /assets/(id)/kpis over that single day returns 180.0. Neither source said 180: one said
100 and the other said 80. Verified against a development database on main.

The same total is what the chart shows the sum of, because it draws one series per source — so the
KPI is consistent with the chart, but the number is not a quantity anyone asked for. It is a total
across sources, where each source is a separate claim about the same event, not a separate
contribution to it.

Why it happens

get_kpis (flexmeasures/api/v3_0/assets.py) reads:

beliefs = sensor.search_beliefs(
    event_starts_after=start,
    event_ends_before=end,
    most_recent_beliefs_only=True,
)

most_recent_beliefs_only is per source, so this returns one row per event per source. The
default use_latest_version_per_event=True does collapse sources sharing a (name, type, model),
but only within the same belief time, unless one_deterministic_belief_per_event is set — which the
KPI query does not set. Two sources believing the same event at different moments therefore both
survive into values, and sum adds them.

Is it intended?

Partly, and that is what makes it worth a decision rather than a quick patch. The code comment above
that call says the opposite of what the call does:

# The beliefs the chart draws: one value per event, the most recent one.
# Aggregating belief rows instead would count a revision on top of what it revised,
# and would count each source separately when several report the same sensor.

The second line describes exactly what still happens across sources. Meanwhile
test_kpi_reports_what_the_chart_draws deliberately asserts a total across two sources — but there
each source reports a different event, so the sum is meaningful. The case above, two sources on the
same event, is not covered by any test.

So the principle "a KPI must describe the same beliefs as the chart" is right; what is missing is
what a KPI should do when the chart draws two points for one event.

Possible directions

  1. Pick one source per event. Pass one_deterministic_belief_per_event=True in the KPI query,
    which keeps the highest-priority source per event (by version, then id). One line, and it matches
    the comment's intent. It changes what a KPI reports for sensors that legitimately carry several
    sources today, and it makes the KPI stop agreeing with a chart that still draws both.
  2. Let the KPI say which source it means, via a source key in the sensors_to_show_as_kpis
    entry, defaulting to today's behaviour. Explicit, but it asks the user to know about sources.
  3. Leave the total as is and make the chart and the KPI agree visibly — for instance by having a
    multi-source KPI say so, rather than presenting one number as if it were unambiguous.

I would lean to 1 for sum in particular, since summing two claims about one event has no reading
under which it is the right answer, whereas mean, min and max at least degrade gracefully.

How to reproduce

Add to flexmeasures/api/v3_0/tests/test_assets_api.py, beside test_kpi_reports_what_the_chart_draws:

@pytest.mark.parametrize("requesting_user", ["test_admin_user@seita.nl"], indirect=True)
def test_kpi_two_sources_same_event(
    db, client, setup_api_test_data, setup_sources, requesting_user
):
    asset_type = db.session.query(GenericAssetType).filter_by(name="battery").one_or_none()
    asset = GenericAsset(
        name="kpi two sources same event",
        generic_asset_type=asset_type,
        account_id=requesting_user.account_id,
    )
    db.session.add(asset)
    db.session.flush()
    sensor = Sensor(
        name="kpi two sources sensor",
        generic_asset=asset,
        event_resolution=timedelta(days=1),
        unit="EUR",
    )
    db.session.add(sensor)
    db.session.flush()
    sources = list(setup_sources.values())
    a, b = sources[0], sources[-1]
    window_start = datetime(2030, 3, 15, tzinfo=utc)
    db.session.bulk_insert_mappings(
        TimedBelief,
        [
            dict(event_start=window_start, belief_horizon=timedelta(days=2),
                 event_value=100.0, sensor_id=sensor.id, source_id=a.id,
                 cumulative_probability=0.5),
            dict(event_start=window_start, belief_horizon=timedelta(days=1),
                 event_value=80.0, sensor_id=sensor.id, source_id=b.id,
                 cumulative_probability=0.5),
        ],
    )
    asset.sensors_to_show_as_kpis = [
        {"title": "Daily costs", "sensor": sensor.id, "function": "sum"}
    ]
    db.session.flush()
    total = _kpi_total(
        client, asset, window_start.isoformat(),
        (window_start + timedelta(days=1)).isoformat(),
    )
    assert total == pytest.approx(180.0)  # what happens today

Why now

This is pre-existing behaviour, and reproduces on main independently of any open branch. It comes up
now because PR #2464 makes a scheduled sensor able to carry schedules from several sources: after a
change to an asset's flex config, the sensor holds the schedule computed under each configuration.
Summing two schedules for the same period is meaningless in exactly the way described above, so the
window in which this bites gets wider.

Raised out of the discussion on #2464.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    UIbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions