diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst index e2c4a1283a..5c41fed611 100644 --- a/documentation/cli/change_log.rst +++ b/documentation/cli/change_log.rst @@ -17,6 +17,8 @@ since v1.0.0 | August 11, 2026 * Add ``flexmeasures jobs run-automations`` to queue jobs for all automations that are due to run this minute from standard five-field cron expressions. Run this command once per minute. It makes at most one queueing attempt per automation per minute, including when an attempt fails after partially queueing jobs. Runs missed while the runner was down are caught up once, with several missed runs coalesced into the latest useful one, and a run at a skipped or repeated daylight-saving-time hour happens exactly once. * ``flexmeasures delete sensor`` now warns which automations read from or write to a sensor before it is deleted, as an automation refers to its sensors by ID and would fail on its next run. * ``flexmeasures show data-sources`` now shows the account a data source belongs to, and lists the sensors holding data recorded by a single source with ``--show-sensors``. +* Add ``flexmeasures show report-templates`` to list prepared report templates, or print one in full (with ``--name``). +* Add a ``--template`` option to ``flexmeasures add report`` and ``flexmeasures add automation`` (type ``reports``), to start from a prepared report template (any ``--config``/``--parameters`` files and other options override it). since v0.33.0 | June 01, 2026 ================================= diff --git a/documentation/cli/commands.rst b/documentation/cli/commands.rst index 205b9c41d6..b06a1f2851 100644 --- a/documentation/cli/commands.rst +++ b/documentation/cli/commands.rst @@ -59,6 +59,7 @@ of which some are referred to in this documentation. ``flexmeasures show data-sources`` List available data sources. ``flexmeasures show beliefs`` Plot time series data. ``flexmeasures show reporters`` List available reporters. +``flexmeasures show report-templates`` List prepared report templates, or print one in full. ``flexmeasures show schedulers`` List available schedulers. ``flexmeasures show chart`` Export charts to PNG or SVG. ``flexmeasures show forecasters`` List available forecasters. diff --git a/documentation/features/reporting.rst b/documentation/features/reporting.rst index 15764192d9..d631e81a99 100644 --- a/documentation/features/reporting.rst +++ b/documentation/features/reporting.rst @@ -148,3 +148,53 @@ For example, this automation computes a report over each past day, every morning flexmeasures add automation --asset 3 --name "Daily aggregation report" --cron "0 1 * * *" --type reports \ --reporter PandasReporter --config reporter-config.yml --parameters report-parameters.yml + + +.. _report_templates: + +Report templates +-------------------- + +FlexMeasures ships with prepared report templates: ready-made report definitions (a reporter class, a complete reporter config and a parameters skeleton), +so you don't have to author a report definition from scratch. List them with: + +.. code-block:: bash + + $ flexmeasures show report-templates + + Name Reporter Description + ---------------- -------------------- --------------------------------------------------------------------------------------------------------------- + energy-costs ProfitOrLossReporter Energy costs over the reporting window, from a power/energy sensor and a consumption price sensor (costs are positive). + self-consumption PandasReporter Share of produced energy consumed on-site, from a production and a consumption sensor. + +Print a template in full (e.g. to pipe it to a file and edit it): + +.. code-block:: bash + + $ flexmeasures show report-templates --name self-consumption > self-consumption.yml + +In a template's parameters skeleton, you fill in your own sensors by replacing the ``FILL_IN`` placeholders +(a clear validation error points out any placeholders you left unfilled). +The templates also recommend a rolling reporting window (``start-offset``/``end-offset`` fields, reporting on the previous day), for recurring use. + +You can pass a template directly to ``flexmeasures add report`` or ``flexmeasures add automation --type reports`` with the ``--template`` option. +The template then acts as defaults: an explicitly given ``--reporter`` and any top-level keys in your ``--config``/``--parameters`` files override it, +and if you provide any timing fields yourself (``start``/``end``/offsets, in the parameters or as CLI options), the template's recommended timing fields are dropped. +Do not combine ``--template`` with ``--source``: an existing source already determines the reporter and its stored configuration. +The self-consumption ratio is undefined for a reporting period without production, so the template leaves that period's value missing instead of reporting 0% or 100%. +For example, this sets up a daily self-consumption report in which only the sensors needed to be filled in: + +.. code-block:: bash + + $ echo " + input: + - name: production + sensor: 1 + - name: consumption + sensor: 2 + output: + - name: self-consumption + sensor: 3 + " > parameters.yml + $ flexmeasures add automation --asset 3 --name "Daily self-consumption report" \ + --cron "0 1 * * *" --type reports --template self-consumption --parameters parameters.yml diff --git a/flexmeasures/cli/data_add.py b/flexmeasures/cli/data_add.py index c2cf22b215..ec322937d9 100755 --- a/flexmeasures/cli/data_add.py +++ b/flexmeasures/cli/data_add.py @@ -53,6 +53,13 @@ add_default_asset_types, ) from flexmeasures.data.services.automations import create_automation +from flexmeasures.data.services.report_templates import ( + PLACEHOLDER, + find_placeholders, + get_report_template, + list_report_templates, + merge_template_parameters, +) from flexmeasures.data.services.data_sources import ( get_or_create_source, get_data_generator, @@ -1512,6 +1519,63 @@ def _assemble_forecaster_config_and_parameters( return config, parameters +def _apply_report_template( + template_name: str, + reporter_class: str | None, + config: dict, + parameters: dict, + user_provided_timing: bool = False, +) -> tuple[str | None, dict, dict]: + """Use a prepared report template as defaults for the reporter class, config and parameters. + + Anything the user provided wins: an explicitly given reporter class is kept, + and top-level config/parameters keys override the template's. + """ + template = get_report_template(template_name) + if template is None: + available_templates = ", ".join(t["name"] for t in list_report_templates()) + click.secho( + f"Unknown report template '{template_name}'." + f" Available report templates: {available_templates}.", + **MsgStyle.ERROR, + ) + raise click.Abort() + reporter_class = reporter_class or template.get("reporter") + config = {**(template.get("config") or {}), **config} + parameters = merge_template_parameters( + template.get("parameters") or {}, + parameters, + user_provided_timing=user_provided_timing, + ) + return reporter_class, config, parameters + + +def _reject_source_with_report_template( + source: DataSource | None, template_name: str | None +) -> None: + """Reject two competing sources of reporter identity and configuration.""" + if source is not None and template_name is not None: + raise click.UsageError( + "--source cannot be combined with --template: a source already determines" + " the reporter and its configuration. Omit --source to use the prepared template." + ) + + +def _abort_on_unfilled_placeholders(config: dict, parameters: dict): + """Abort with a clear validation error if template placeholders were left unfilled.""" + placeholders = [f"config: {path}" for path in find_placeholders(config)] + [ + f"parameters: {path}" for path in find_placeholders(parameters) + ] + if placeholders: + click.secho( + f"Invalid report definition: replace the '{PLACEHOLDER}' template placeholders" + " with your own values (usually sensor IDs) in the following fields:\n- " + + "\n- ".join(placeholders), + **MsgStyle.ERROR, + ) + raise click.Abort() + + @fm_add_data.command("forecasts") @click.option( "--resolution", @@ -1738,6 +1802,16 @@ def add_forecast( # noqa: C901 help="Reporter class registered in flexmeasures.data.models.reporting or in an available flexmeasures plugin (only used for --type reports)." " Use the command `flexmeasures show reporters` to list all the available reporters.", ) +@click.option( + "--template", + "template_name", + required=False, + type=click.STRING, + help="Name of a prepared report template to use as defaults for the reporter, config and parameters" + " (only used for --type reports). Any --config/--parameters files and other options override it." + " Cannot be combined with --source, which already determines the reporter and configuration." + " Use the command `flexmeasures show report-templates` to list all the available templates.", +) @click.option( "--source", "source", @@ -1778,6 +1852,7 @@ def add_automation( inactive: bool = False, forecaster_class: str | None = None, reporter_class: str | None = None, + template_name: str | None = None, source: DataSource | None = None, config_file: TextIOBase | None = None, parameters_file: TextIOBase | None = None, @@ -1794,8 +1869,8 @@ def add_automation( flexmeasures add automation --asset 3 --name "Hourly schedules" --cron "0 * * * *" --type schedules --parameters trigger-message.yml flexmeasures add automation --asset 3 --name "Daily self-consumption report" - --cron "0 1 * * *" --type reports --reporter PandasReporter - --config reporter-config.yml --parameters report-parameters.yml + --cron "0 1 * * *" --type reports --template self-consumption + --parameters report-parameters.yml For forecasts and reports, the data generator configuration is stored on a data source, and the parameters are validated and stored on the automation itself. @@ -1820,10 +1895,24 @@ def add_automation( if forecaster_class is None: forecaster_class = "TrainPredictPipeline" + if template_name is not None and automation_type != "reports": + click.secho( + "The --template option is only supported for report automations (--type reports).", + **MsgStyle.ERROR, + ) + raise click.Abort() + _reject_source_with_report_template(source, template_name) + config, parameters = _assemble_forecaster_config_and_parameters( kwargs, source, config_file, parameters_file ) + if template_name is not None: + reporter_class, config, parameters = _apply_report_template( + template_name, reporter_class, config, parameters + ) + if automation_type == "reports": + _abort_on_unfilled_placeholders(config, parameters) if automation_type == "schedules": # Only options actually given on the command line count: the forecaster and the # configuration options that were left out still show up here, with their defaults. @@ -2083,11 +2172,22 @@ def add_schedule( # noqa C901 @click.option( "--reporter", "reporter_class", - default="PandasReporter", + default=None, type=click.STRING, - help="Reporter class registered in flexmeasures.data.models.reporting or in an available flexmeasures plugin." + help="Reporter class registered in flexmeasures.data.models.reporting or in an available flexmeasures plugin" + " (defaults to PandasReporter, or to the template's reporter if --template is given)." " Use the command `flexmeasures show reporters` to list all the available reporters.", ) +@click.option( + "--template", + "template_name", + required=False, + type=click.STRING, + help="Name of a prepared report template to use as defaults for the reporter, config and parameters." + " Any --config/--parameters files and other options override it." + " Cannot be combined with --source, which already determines the reporter and configuration." + " Use the command `flexmeasures show report-templates` to list all the available templates.", +) @click.option( "--start", "start", @@ -2171,7 +2271,8 @@ def add_schedule( # noqa C901 "To process the job, run a worker (on any computer, but configured to the same databases) to process the 'reporting' queue. Defaults to False.", ) def add_report( # noqa: C901 - reporter_class: str, + reporter_class: str | None = None, + template_name: str | None = None, source: DataSource | None = None, config_file: TextIOBase | None = None, parameters_file: TextIOBase | None = None, @@ -2192,6 +2293,8 @@ def add_report( # noqa: C901 Create a new report using the Reporter class and save the results to the database or export them as CSV or Excel file. """ + _reject_source_with_report_template(source, template_name) + if as_job and (dry_run or output_file_pattern): click.secho( "The --as-job flag cannot be combined with --dry-run or --output-file:" @@ -2210,18 +2313,46 @@ def add_report( # noqa: C901 config = dict() if config_file: - config = yaml.safe_load(config_file) + config = _load_yaml_mapping(config_file, "--config") if edit_config: - config = launch_editor("/tmp/config.yml") + config = _normalize_yaml_mapping( + launch_editor("/tmp/config.yml"), "--edit-config" + ) parameters = dict() if parameters_file: - parameters = yaml.safe_load(parameters_file) + parameters = _load_yaml_mapping(parameters_file, "--parameters") if edit_parameters: - parameters = launch_editor("/tmp/parameters.yml") + parameters = _normalize_yaml_mapping( + launch_editor("/tmp/parameters.yml"), "--edit-parameters" + ) + + if template_name is not None: + reporter_class, config, parameters = _apply_report_template( + template_name, + reporter_class, + config, + parameters, + user_provided_timing=any( + timing_option is not None + for timing_option in (start, end, start_offset, end_offset) + ), + ) + if reporter_class is None: + reporter_class = "PandasReporter" + + # Offset fields in the parameters (e.g. from a template) serve as fallbacks for the CLI options + start_offset = ( + start_offset if start_offset is not None else parameters.get("start-offset") + ) + end_offset = end_offset if end_offset is not None else parameters.get("end-offset") + parameters.pop("start-offset", None) + parameters.pop("end-offset", None) + + _abort_on_unfilled_placeholders(config, parameters) # check if sensor is not provided in the `parameters` description if "output" not in parameters or len(parameters["output"]) == 0: diff --git a/flexmeasures/cli/data_show.py b/flexmeasures/cli/data_show.py index 695ac09770..044f761575 100644 --- a/flexmeasures/cli/data_show.py +++ b/flexmeasures/cli/data_show.py @@ -950,6 +950,56 @@ def list_reporters(): list_data_generators("reporter") +@fm_show_data.command("report-templates") +@with_appcontext +@click.option( + "--name", + "name", + required=False, + type=click.STRING, + help="Print the full YAML of the template with this name (e.g. to pipe it to a file).", +) +def show_report_templates(name: str | None = None): + """ + Show prepared report templates: ready-made report definitions to start from. + + Fill in your own sensors (replacing the FILL_IN placeholders) and pass a template + to `flexmeasures add report` or `flexmeasures add automation --type reports` + using the --template option. + """ + from flexmeasures.data.services.report_templates import ( + get_report_template_text, + list_report_templates, + ) + + if name is not None: + template_text = get_report_template_text(name) + if template_text is None: + click.secho( + f"Unknown report template '{name}'." + " Use `flexmeasures show report-templates` to list the available templates.", + **MsgStyle.ERROR, + ) + raise click.Abort() + click.echo(template_text) + return + + click.echo("Prepared report templates:\n") + click.echo( + tabulate( + [ + (template["name"], template["reporter"], template["description"]) + for template in list_report_templates() + ], + headers=["Name", "Reporter", "Description"], + ) + ) + click.echo( + "\nUse `flexmeasures show report-templates --name ` to print a template in full," + "\nor pass `--template ` to `flexmeasures add report` or `flexmeasures add automation --type reports`." + ) + + @fm_show_data.command("schedulers") @with_appcontext def list_schedulers(): diff --git a/flexmeasures/cli/tests/test_report_templates.py b/flexmeasures/cli/tests/test_report_templates.py new file mode 100644 index 0000000000..e8a2a687cb --- /dev/null +++ b/flexmeasures/cli/tests/test_report_templates.py @@ -0,0 +1,355 @@ +"""Tests for the prepared report templates and their CLI integration.""" + +from copy import deepcopy +from datetime import timedelta + +import pandas as pd +import pytest +import yaml + +from sqlalchemy import select, update + +from flexmeasures.data.models.data_sources import DataSource +from flexmeasures.data.models.automations import Automation +from flexmeasures.data.models.generic_assets import GenericAsset +from flexmeasures.data.models.reporting.pandas_reporter import PandasReporter +from flexmeasures.data.models.time_series import Sensor, TimedBelief +from flexmeasures.data.services.automations import prepare_report_parameters +from flexmeasures.data.services.report_templates import ( + find_placeholders, + get_report_template, + list_report_templates, +) + + +@pytest.fixture(scope="function") +def clean_redis(app): + app.redis_connection.flushdb() + yield + app.redis_connection.flushdb() + + +def _fill_sensors( + parameters: dict, input_sensor_ids: list[int], output_sensor_ids: list[int] +) -> dict: + """Fill the sensor placeholders in a template's parameters skeleton.""" + parameters = deepcopy(parameters) + for description, sensor_id in zip(parameters["input"], input_sensor_ids): + description["sensor"] = sensor_id + for description, sensor_id in zip(parameters["output"], output_sensor_ids): + description["sensor"] = sensor_id + return parameters + + +def test_energy_costs_template_validates(app, fresh_db, setup_dummy_asset): + """The energy-costs template validates against the ProfitOrLossReporter schemas, once sensors are filled in.""" + template = get_report_template("energy-costs") + assert template["reporter"] == "ProfitOrLossReporter" + + asset = fresh_db.session.get(GenericAsset, setup_dummy_asset) + price_sensor = Sensor( + "price", + generic_asset=asset, + event_resolution=timedelta(hours=1), + unit="EUR/MWh", + ) + power_sensor = Sensor( + "power", generic_asset=asset, event_resolution=timedelta(hours=1), unit="MW" + ) + cost_sensor = Sensor( + "costs", generic_asset=asset, event_resolution=timedelta(days=1), unit="EUR" + ) + fresh_db.session.add_all([price_sensor, power_sensor, cost_sensor]) + fresh_db.session.flush() + + reporter_class = app.data_generators["reporter"][template["reporter"]] + + # the config loads cleanly, once the price sensor placeholder is filled in + config = dict(template["config"], consumption_price_sensor=price_sensor.id) + assert find_placeholders(config) == [] + reporter_class._config_schema.load(config) + + # the parameters skeleton loads cleanly, once the sensor placeholders are + # filled in and the recommended rolling window is resolved + parameters = _fill_sensors( + template["parameters"], [power_sensor.id], [cost_sensor.id] + ) + assert find_placeholders(parameters) == [] + prepared_parameters = prepare_report_parameters(parameters, "0 1 * * *") + reporter_class._parameters_schema.load(prepared_parameters) + + +def test_self_consumption_template_validates(app, fresh_db, setup_dummy_data): + """The self-consumption template validates against the PandasReporter schemas, once sensors are filled in.""" + template = get_report_template("self-consumption") + assert template["reporter"] == "PandasReporter" + + reporter_class = app.data_generators["reporter"][template["reporter"]] + + # the config is complete and valid as-is (sensors only enter through the parameters) + assert find_placeholders(template["config"]) == [] + reporter_class._config_schema.load(template["config"]) + + sensor1_id, sensor2_id, report_sensor_id, _ = setup_dummy_data + parameters = _fill_sensors( + template["parameters"], [sensor1_id, sensor2_id], [report_sensor_id] + ) + assert find_placeholders(parameters) == [] + prepared_parameters = prepare_report_parameters(parameters, "0 1 * * *") + reporter_class._parameters_schema.load(prepared_parameters) + + +def test_show_report_templates(app): + """The show command lists all packaged templates, and prints a single template in full.""" + from flexmeasures.cli.data_show import show_report_templates + + runner = app.test_cli_runner() + + result = runner.invoke(show_report_templates) + assert result.exit_code == 0, result.output + for name, reporter in [ + ("energy-costs", "ProfitOrLossReporter"), + ("self-consumption", "PandasReporter"), + ]: + assert name in result.output + assert reporter in result.output + + result = runner.invoke(show_report_templates, ["--name", "self-consumption"]) + assert result.exit_code == 0, result.output + assert "PandasReporter" in result.output + assert "FILL_IN" in result.output + # the printed YAML can be piped to a file and loads cleanly + assert yaml.safe_load(result.output)["name"] == "self-consumption" + + result = runner.invoke(show_report_templates, ["--name", "unknown"]) + assert result.exit_code != 0 + assert "Unknown report template" in result.output + + +def test_add_and_run_report_automation_with_template( + app, fresh_db, setup_dummy_data, clean_redis, tmp_path +): + """A report automation created from the self-consumption template computes a working report.""" + from flexmeasures.cli.data_add import add_automation + from flexmeasures.cli.jobs import run_automations + from flexmeasures.utils.job_utils import work_on_rq + + sensor1_id, sensor2_id, report_sensor_id, _ = setup_dummy_data + report_sensor = fresh_db.session.get(Sensor, report_sensor_id) + daily_sensor = Sensor( + "daily self-consumption", + generic_asset=report_sensor.generic_asset, + event_resolution=timedelta(days=1), + ) + fresh_db.session.add(daily_sensor) + fresh_db.session.commit() + daily_sensor_id = daily_sensor.id + + # Fill in the template's sensor placeholders, and use an absolute reporting window + # (the dummy data lives in April 2023), replacing the template's rolling window + parameters = dict( + input=[ + dict(name="production", sensor=sensor1_id), + dict(name="consumption", sensor=sensor2_id), + ], + output=[dict(name="self-consumption", sensor=daily_sensor_id)], + start="2023-04-10T00:00:00+00:00", + end="2023-04-12T00:00:00+00:00", + ) + parameters_file = tmp_path / "parameters.yml" + parameters_file.write_text(yaml.dump(parameters)) + + runner = app.test_cli_runner() + result = runner.invoke( + add_automation, + [ + "--asset", str(daily_sensor.generic_asset_id), + "--name", "Daily self-consumption", + "--cron", "* * * * *", # due every minute + "--type", "reports", + "--template", "self-consumption", + "--parameters", str(parameters_file), + ], + ) # fmt: skip + assert "Successfully created" in result.output, result.output + + automation = fresh_db.session.execute(select(Automation)).scalar_one() + assert automation.type == "reports" + assert automation.generator is not None + assert automation.generator.model == "PandasReporter" + # the reporter config came from the template + template = get_report_template("self-consumption") + stored_config = automation.generator.attributes["data_generator"]["config"] + assert stored_config["required_input"] == template["config"]["required_input"] + assert len(stored_config["transformations"]) == len( + template["config"]["transformations"] + ) + # user-provided timing fields replaced the template's rolling window + assert "start-offset" not in automation.parameters + assert "end-offset" not in automation.parameters + assert automation.parameters["start"] == "2023-04-10T00:00:00+00:00" + + # run the automation and process the queued reporting job + result = runner.invoke(run_automations) + assert result.exit_code == 0, result.output + assert "queued 1 reporting job(s)" in result.output, result.output + work_on_rq(app.queues["reporting"]) + + stored_report = fresh_db.session.get(Sensor, daily_sensor_id).search_beliefs( + event_starts_after="2023-04-10T00:00:00+00:00", + event_ends_before="2023-04-12T00:00:00+00:00", + ) + # both sensors hold identical data, so all production is consumed on-site + assert len(stored_report) == 2 + assert (stored_report.values.T == [1.0, 1.0]).all() + + +def test_report_template_placeholders_must_be_filled(app, fresh_db, setup_dummy_data): + """Leaving template placeholders unfilled produces a clear validation error.""" + from flexmeasures.cli.data_add import add_automation, add_report + + runner = app.test_cli_runner() + + # add automation: unfilled sensor placeholders are rejected with a clear error + result = runner.invoke( + add_automation, + [ + "--asset", "1", + "--name", "Unfilled report", + "--cron", "0 1 * * *", + "--type", "reports", + "--template", "self-consumption", + ], + ) # fmt: skip + assert result.exit_code != 0 + assert "FILL_IN" in result.output + assert "parameters: input[0].sensor" in result.output + + # add report: same + result = runner.invoke(add_report, ["--template", "energy-costs"]) + assert result.exit_code != 0 + assert "FILL_IN" in result.output + assert "config: consumption_price_sensor" in result.output + + # unknown templates are rejected, listing the available ones + result = runner.invoke( + add_automation, + [ + "--asset", "1", + "--name", "Unknown template", + "--cron", "0 1 * * *", + "--type", "reports", + "--template", "unknown", + ], + ) # fmt: skip + assert result.exit_code != 0 + assert "Unknown report template" in result.output + for template in list_report_templates(): + assert template["name"] in result.output + + # templates are only supported for report automations + result = runner.invoke( + add_automation, + [ + "--asset", "1", + "--name", "Forecasts from a report template", + "--cron", "0 1 * * *", + "--template", "self-consumption", + ], + ) # fmt: skip + assert result.exit_code != 0 + assert "only supported for report automations" in result.output + + +def test_report_template_cannot_be_combined_with_source( + app, fresh_db, setup_dummy_data +): + """A stored reporter source and a report template cannot both define the reporter configuration.""" + from flexmeasures.cli.data_add import add_automation, add_report + + source_id = fresh_db.session.execute(select(DataSource.id).limit(1)).scalar_one() + runner = app.test_cli_runner() + + result = runner.invoke( + add_report, + ["--source", str(source_id), "--template", "self-consumption"], + ) + assert result.exit_code != 0 + assert "--source cannot be combined with --template" in result.output + + result = runner.invoke( + add_automation, + [ + "--asset", + "1", + "--name", + "Ambiguous report configuration", + "--type", + "reports", + "--source", + str(source_id), + "--template", + "self-consumption", + ], + ) + assert result.exit_code != 0 + assert "--source cannot be combined with --template" in result.output + + +@pytest.mark.parametrize("option", ["--config", "--parameters"]) +def test_add_report_template_rejects_non_mapping_override( + app, fresh_db, tmp_path, option +): + """Template override files must be mappings instead of producing an internal merge error.""" + from flexmeasures.cli.data_add import add_report + + override_file = tmp_path / "override.yaml" + override_file.write_text("- not\n- a\n- mapping\n") + + result = app.test_cli_runner().invoke( + add_report, + ["--template", "self-consumption", option, str(override_file)], + ) + + assert result.exit_code != 0 + assert f"The {option} file must contain a YAML or JSON object" in result.output + + +def test_self_consumption_is_undefined_without_production( + app, fresh_db, setup_dummy_data +): + """The self-consumption ratio is missing when a reporting period has no production.""" + production_id, consumption_id, report_sensor_id, _ = setup_dummy_data + report_sensor = fresh_db.session.get(Sensor, report_sensor_id) + daily_sensor = Sensor( + "daily self-consumption with zero production", + generic_asset=report_sensor.generic_asset, + event_resolution=timedelta(days=1), + ) + fresh_db.session.add(daily_sensor) + fresh_db.session.execute( + update(TimedBelief) + .where( + TimedBelief.sensor_id == production_id, + TimedBelief.event_start < "2023-04-11T00:00:00+00:00", + ) + .values(event_value=0) + ) + fresh_db.session.commit() + + template = get_report_template("self-consumption") + report = PandasReporter(config=template["config"]).compute( + parameters={ + "input": [ + {"name": "production", "sensor": production_id}, + {"name": "consumption", "sensor": consumption_id}, + ], + "output": [{"name": "self-consumption", "sensor": daily_sensor.id}], + "start": "2023-04-10T00:00:00+00:00", + "end": "2023-04-12T00:00:00+00:00", + } + )[0]["data"] + + assert len(report) == 2 + assert pd.isna(report.values[0, 0]) + assert report.values[1, 0] == pytest.approx(1) diff --git a/flexmeasures/data/services/report_templates.py b/flexmeasures/data/services/report_templates.py new file mode 100644 index 0000000000..96e0329831 --- /dev/null +++ b/flexmeasures/data/services/report_templates.py @@ -0,0 +1,93 @@ +""" +Logic for loading prepared report templates. + +Report templates are YAML files packaged in flexmeasures/data/templates/reports. +Each template describes a ready-made report definition: a reporter class, +a complete reporter config and a parameters skeleton in which users fill in +their own sensors (replacing the FILL_IN placeholders). +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import yaml + +TEMPLATES_DIR = Path(__file__).resolve().parent.parent / "templates" / "reports" + +# Placeholder value users need to replace (usually with one of their sensor IDs) +PLACEHOLDER = "FILL_IN" + +# Fields (within report parameters) that determine the reporting window +TIMING_FIELDS = ("start", "end", "start-offset", "end-offset") + + +def _template_paths() -> list[Path]: + return sorted(TEMPLATES_DIR.glob("*.y*ml")) + + +def _template_path(name: str) -> Path | None: + for path in _template_paths(): + if path.stem == name: + return path + return None + + +def list_report_templates() -> list[dict]: + """Load all packaged report templates (sorted by name).""" + return [yaml.safe_load(path.read_text()) for path in _template_paths()] + + +def get_report_template(name: str) -> dict | None: + """Load the packaged report template with the given name, if it exists.""" + path = _template_path(name) + if path is None: + return None + return yaml.safe_load(path.read_text()) + + +def get_report_template_text(name: str) -> str | None: + """The raw YAML of the packaged report template with the given name, if it exists. + + Unlike `get_report_template`, this preserves the explanatory comments, + so users can pipe the result to a file and edit it. + """ + path = _template_path(name) + if path is None: + return None + return path.read_text() + + +def merge_template_parameters( + template_parameters: dict, + user_parameters: dict, + user_provided_timing: bool = False, +) -> dict: + """Merge user-provided report parameters on top of a template's parameters skeleton. + + Top-level keys provided by the user win. Timing fields are treated as a group: + if the user provides any timing field (or `user_provided_timing` is set, e.g. + because timing was given through CLI options), the template's recommended + timing fields are dropped altogether. + """ + merged = dict(template_parameters) + if user_provided_timing or any(field in user_parameters for field in TIMING_FIELDS): + for field in TIMING_FIELDS: + merged.pop(field, None) + merged.update(user_parameters) + return merged + + +def find_placeholders(obj: Any, root: str = "") -> list[str]: + """List the paths of any unfilled template placeholders in the given (nested) object.""" + paths: list[str] = [] + if isinstance(obj, dict): + for k, v in obj.items(): + paths += find_placeholders(v, f"{root}.{k}" if root else str(k)) + elif isinstance(obj, list): + for i, v in enumerate(obj): + paths += find_placeholders(v, f"{root}[{i}]") + elif isinstance(obj, str) and obj == PLACEHOLDER: + paths.append(root) + return paths diff --git a/flexmeasures/data/templates/reports/energy-costs.yaml b/flexmeasures/data/templates/reports/energy-costs.yaml new file mode 100644 index 0000000000..a5398130eb --- /dev/null +++ b/flexmeasures/data/templates/reports/energy-costs.yaml @@ -0,0 +1,30 @@ +# Prepared report template: energy costs. +# +# Computes the costs of consumed energy (and any revenues from produced energy) +# over the reporting window, using the ProfitOrLossReporter. +# Replace each FILL_IN placeholder with one of your own sensor IDs. +name: energy-costs +description: Energy costs over the reporting window, from a power/energy sensor and a consumption price sensor (costs are positive). +reporter: ProfitOrLossReporter +config: + # The sensor recording the price of consumed energy (e.g. in EUR/MWh). + consumption_price_sensor: FILL_IN + # Optionally, set a separate feed-in tariff (defaults to the consumption price): + # production_price_sensor: FILL_IN + # Report costs as positive values (and revenues as negative values). + loss_is_positive: true +parameters: + input: + # The power or energy sensor to compute the costs for. By default, positive + # values denote production and negative values denote consumption (set the + # sensor attribute consumption_is_positive to invert this). + - sensor: FILL_IN + output: + # The sensor to store the report; its unit must be a currency (e.g. EUR). + # The results are resampled to this sensor's event resolution (e.g. daily costs). + - sensor: FILL_IN + # Rolling reporting window, recommended for recurring use (e.g. in an automation): + # each run reports on the previous day. Replace with fixed "start" and "end" + # datetimes for a one-off report. + start-offset: -1D,DB + end-offset: DB diff --git a/flexmeasures/data/templates/reports/self-consumption.yaml b/flexmeasures/data/templates/reports/self-consumption.yaml new file mode 100644 index 0000000000..2179ff23eb --- /dev/null +++ b/flexmeasures/data/templates/reports/self-consumption.yaml @@ -0,0 +1,54 @@ +# Prepared report template: self-consumption. +# +# Computes the share of produced energy that is consumed on-site, +# using the PandasReporter with a production and a consumption sensor. +# Replace each FILL_IN placeholder with one of your own sensor IDs. +name: self-consumption +description: Share of produced energy consumed on-site, from a production and a consumption sensor. +reporter: PandasReporter +config: + required_input: + # The production and consumption sensors should record positive values in the same unit. + - name: production + - name: consumption + required_output: + - name: self-consumption + transformations: + # Production consumed on-site: the elementwise minimum of production and consumption. + - df_input: production + df_output: self-consumption + method: clip + kwargs: + upper: "@consumption" + # Average both over the reporting periods + # (edit "1D" to match the event resolution of your output sensor). + - method: resample_events + args: ["1D"] + df_output: self-consumption + - df_input: production + df_output: total-production + method: resample_events + args: ["1D"] + # The share of production consumed on-site: a fraction between 0 and 1. + - df_input: self-consumption + df_output: self-consumption + method: div + args: ["@total-production"] + # If total production is zero, the ratio is undefined and remains missing. +parameters: + input: + # The sensor recording produced power/energy (positive values). + - name: production + sensor: FILL_IN + # The sensor recording consumed power/energy (positive values). + - name: consumption + sensor: FILL_IN + output: + # The sensor to store the report (a dimensionless fraction between 0 and 1). + - name: self-consumption + sensor: FILL_IN + # Rolling reporting window, recommended for recurring use (e.g. in an automation): + # each run reports on the previous day. Replace with fixed "start" and "end" + # datetimes for a one-off report. + start-offset: -1D,DB + end-offset: DB