Automations - first roundtrip for forecasts

documentation/api/change_log.rst

@@ -5,8 +5,13 @@ API change log
 
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL (e.g. `/api/v3_0`), allowing developers to upgrade at their own pace.
 
+v3.0-33 | September 1, 2026
+"""""""""""""""""""""""""""
+- Added ``GET /api/v3_0/assets/<id>/automations`` and ``GET /api/v3_0/assets/<id>/automations/<automation_id>`` for listing and inspecting forecast automations, including the sensors an automation reads from and writes to. Each automation shows the IANA ``timezone`` in which its cron expression is interpreted, and a ``cursor``: the offset-aware UTC time of the most recent run it committed to. The cursor advances just before queueing, so it does not indicate that queueing or the forecast itself succeeded. Asset job entries now include ``created_via`` provenance; automation identity is included only when the caller may read that automation.
+- Added ``GET /api/v3_0/sources/<id>`` to show the full record of one data source, including the attributes in which data generators store their configuration.
+
 v3.0-32 | August 11, 2026
-""""""""""""""""""""""""""
+"""""""""""""""""""""""""
 - API endpoints are now rate-limited. A request which exceeds a limit is answered with a ``429 (Too Many Requests)`` status code and a ``Retry-After`` header stating how many seconds to wait. Responses also carry ``X-RateLimit-*`` headers, describing the limit that applied, how much of it is left, and when it resets. A stricter limit applies to ``POST /assets/<id>/schedules/trigger``, ``POST /sensors/<id>/schedules/trigger`` and ``POST /sensors/<id>/forecasts/trigger`` than to other endpoints; the health endpoints are exempt. Per-account overrides are set by assigning the account a plan (a ``Plan`` database row), rather than through an account attribute.
 - Introduced the ``inflexible-consumption`` and ``inflexible-production`` flex-context fields, which make explicit how the sign of each inflexible device's power data should be read: positive values denote consumption resp. production. Each entry is a sensor reference (``{"sensor": <id>}``), optionally with source filters (``source-types``, ``exclude-source-types``, ``sources``, ``source-account``). Deprecated the ``inflexible-device-sensors`` field (a list of bare sensor IDs, whose sign convention is read from each sensor's ``consumption_is_positive`` attribute); it remains supported, but cannot be combined with the new fields in one flex-context.
 - Added a ``role`` query parameter to ``GET /api/v3_0/accounts`` for filtering accessible organisations by account role.

documentation/changelog.rst

@@ -53,6 +53,9 @@ v1.0.0 | August 25, 2026
 New features
 -------------
 
+* Automations - first roundtrip for forecasts: recurring tasks defined per asset, managed with new CLI commands (``flexmeasures add|edit|delete automation``), run by ``flexmeasures jobs run-automations``, and viewable in a new UI page and API endpoints (``[GET] /assets/(id)/automations``); each automation interprets its recurrence in its own timezone, and runs missed while the runner was down are caught up once, coalesced into one current forecast; an automation's details link to the sensors it reads from and writes to, a sensor's page lists the automations feeding it, and deleting a sensor warns about the automations that use it; jobs now also record whether they were created via the CLI, the API or an automation [see `PR #2290 <https://www.github.com/FlexMeasures/flexmeasures/pull/2290>`_ and `PR #2396 <https://www.github.com/FlexMeasures/flexmeasures/pull/2396>`_]
+* In the UI, the full record of the data source selected on a sensor page can be inspected, backed by a new API endpoint (``[GET] /sources/(id)``) [see `PR #2290 <https://www.github.com/FlexMeasures/flexmeasures/pull/2290>`_]
+* ``flexmeasures show data-sources`` now shows which organisation a data source belongs to, and can list the sensors holding data recorded by a given source [see `PR #2401 <https://www.github.com/FlexMeasures/flexmeasures/pull/2401>`_]
 * The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #1946 <https://www.github.com/FlexMeasures/flexmeasures/pull/1946>`_, `PR #2172 <https://www.github.com/FlexMeasures/flexmeasures/pull/2172>`_, `PR #2235 <https://www.github.com/FlexMeasures/flexmeasures/pull/2235>`_, `PR #2271 <https://www.github.com/FlexMeasures/flexmeasures/pull/2271>`_, `PR #2355 <https://www.github.com/FlexMeasures/flexmeasures/pull/2355>`_ and `PR #2380 <https://www.github.com/FlexMeasures/flexmeasures/pull/2380>`_]
 * Support multiple feeders to a shared storage [see `PR #2001 <https://www.github.com/FlexMeasures/flexmeasures/pull/2001>`_, `PR #2321 <https://www.github.com/FlexMeasures/flexmeasures/pull/2321>`_, `PR #2322 <https://www.github.com/FlexMeasures/flexmeasures/pull/2322>`_, `PR #2325 <https://www.github.com/FlexMeasures/flexmeasures/pull/2325>`_ and `PR #2431 <https://www.github.com/FlexMeasures/flexmeasures/pull/2431>`_]
 * Add support for intermediate power constraints on groups of devices, via a new ``group`` field in the storage flex-model [see `PR #2276 <https://www.github.com/FlexMeasures/flexmeasures/pull/2276>`_ and `issue #2092 <https://github.com/FlexMeasures/flexmeasures/issues/2092>`_]

documentation/cli/change_log.rst

@@ -12,6 +12,9 @@ since v1.0.0 | August 11, 2026
 * Add ``flexmeasures add plan``, ``flexmeasures show plans`` and ``flexmeasures edit plan``, to manage the rate limits and quotas which apply to the accounts on a plan.
 * Add ``flexmeasures edit secret`` to store an encrypted secret on an account or asset.
 * Add ``flexmeasures delete secret`` to remove an encrypted secret from an account or asset.
+* Add ``flexmeasures add automation``, ``flexmeasures edit automation`` and ``flexmeasures delete automation`` to manage automations (recurring tasks on an asset; for now, computing forecasts). Each automation carries its own IANA timezone (``--timezone``), in which its cron expression is interpreted.
+* Add ``flexmeasures jobs run-automations`` to queue jobs for all automations that are due to run this minute from standard five-field cron expressions. Run this command once per minute. It makes at most one queueing attempt per automation per minute, including when an attempt fails after partially queueing jobs. Runs missed while the runner was down are caught up once, with several missed forecast runs coalesced into the latest useful forecast, and a run at a skipped or repeated daylight-saving-time hour happens exactly once.
+* ``flexmeasures delete sensor`` now warns which automations read from or write to a sensor before it is deleted, as an automation refers to its sensors by ID and would fail on its next run.
 * ``flexmeasures show data-sources`` now shows the account a data source belongs to, and lists the sensors holding data recorded by a single source with ``--show-sensors``.
 
 since v0.33.0 | June 01, 2026

documentation/cli/commands.rst

@@ -41,6 +41,7 @@ of which some are referred to in this documentation.
 ``flexmeasures add annotation``                   Add annotation to accounts, assets and/or sensors.
 ``flexmeasures add toy-account``                  Create a toy account, for tutorials and trying things.
 ``flexmeasures add report``                       Create a report.
+``flexmeasures add automation``                   Add an automation with its own cron timezone.
 ================================================= =======================================
 
 
@@ -75,6 +76,7 @@ of which some are referred to in this documentation.
 ``flexmeasures edit resample-data``               Assign a new event resolution to an existing sensor and resample its data accordingly.
 ``flexmeasures edit transfer-parenthood``         (Re)assign parent assets.
 ``flexmeasures edit transfer-ownership``          Transfer the ownership of an asset and its children to a different account.
+``flexmeasures edit automation``                  Edit an automation's name, recurrence, timezone or activation status.
 ================================================= =======================================
 
 ``delete`` - Delete data
@@ -93,6 +95,7 @@ of which some are referred to in this documentation.
 ``flexmeasures delete prognoses``                 Delete forecasts and schedules (forecasts > 0).
 ``flexmeasures delete unchanged-beliefs``         Delete unchanged beliefs.
 ``flexmeasures delete nan-beliefs``               Delete NaN beliefs.
+``flexmeasures delete automation``                Delete an automation.
 ================================================= =======================================
 
 
@@ -117,6 +120,7 @@ of which some are referred to in this documentation.
 ``flexmeasures jobs run-job``                     Run a single job (useful for debugging it)
 ``flexmeasures jobs inspect-job``                 Inspect a background job and print its current status, result and metadata.
 ``flexmeasures jobs stats``                       Show estimated live statistics of the queueing system.
+``flexmeasures jobs run-automations``             Handle due and missed forecast automation runs (invoke once per minute, e.g. via cron).
 ================================================= =======================================
 
 

documentation/features/automations.rst

@@ -0,0 +1,89 @@
+.. _automations:
+
+Automations
+============
+
+An **automation** is a recurring task defined on an asset.
+For now, an automation computes forecasts; automating schedules and reports is planned.
+
+On each run, the automation queues jobs (so make sure a worker is processing the ``forecasting`` queue, see :ref:`redis-queue`).
+The parameters of the task were stored when the automation was created, and validated with the same schema that the CLI and API use.
+Timing parameters are resolved on each run — for instance, the forecast start defaults to the time the automation runs, so each run produces fresh forecasts.
+
+Creating an automation
+----------------------
+
+Here is how you create an automation in the CLI, asking for daily (at 6 AM) forecasts of sensor 12:
+
+.. code-block:: bash
+
+    flexmeasures add automation --asset 3 --name "Daily PV forecasts" --type forecasts \
+        --cron "0 6 * * *" --timezone Europe/Amsterdam --sensor 12
+
+``--type`` says what the automation computes, and defaults to ``forecasts``.
+The remaining options are the ones the task itself needs: a forecast automation accepts everything `flexmeasures add forecast` accepts, such as ``--forecaster`` to pick the forecaster and ``--config`` to configure it (see :ref:`forecasting`).
+The forecaster and its configuration are stored on a data source, so you can also pass ``--source`` to reuse the data source of an existing forecaster, in which case ``--forecaster`` and ``--config`` (and the individual configuration options) are not needed — the data source already determines them.
+That data source is required while the automation exists, so it cannot be deleted until the automation is removed.
+
+The recurrence is defined by a standard five-field cron string (minute, hour, day of month, month, and day of week), which defaults to ``"0 0 * * *"`` (daily at midnight).
+It is interpreted in the automation's IANA timezone.
+If ``--timezone`` is omitted, the current ``FLEXMEASURES_TIMEZONE`` value is copied to the automation.
+Changing that configuration later does not change existing automations.
+Cron aliases and optional seconds or year fields are not supported.
+
+Automations are active by default (use ``--inactive`` to create them in deactivated state).
+Use ``flexmeasures edit automation`` to rename, re-schedule (``--cron``), change the timezone, activate or deactivate an automation, and ``flexmeasures delete automation`` to remove one.
+These changes are recorded in the asset's audit log.
+
+For forecast automations, the sensor on which forecasts are saved (``sensor-to-save``, falling back to ``sensor``) must belong to the automation's asset or one of its descendants.
+This relationship is checked both when the automation is created and immediately before each run.
+
+Running automations
+-------------------
+
+For automations to actually run, let a cron job execute the following command once per minute:
+
+.. code-block:: bash
+
+    * * * * * flexmeasures jobs run-automations
+
+Each due automation then queues its jobs.
+If the runner misses runs, because it was down or overloaded, it catches up when it resumes: it queues only the latest missed run of each automation, rather than replaying stale ones.
+Timing parameters that default to the run time are resolved when that catch-up run is queued, so it produces a current forecast.
+
+Each scheduled run receives at most one automatic queueing attempt.
+If the process crashes, or queueing fails after creating some jobs, that run is not retried automatically, because a retry could duplicate partial work.
+
+The jobs record how they were created, which is shown on the asset's status page (UI), where recent jobs are listed.
+
+Viewing automations
+-------------------
+
+Automations defined on an asset can be viewed on the asset's *Automations* page in the UI, and listed with the API endpoint `[GET] /assets/(id)/automations <../api/v3_0.html#get--api-v3_0-assets-id-automations>`_.
+An automation's details show the sensors it reads from and writes to, linking to each sensor's page.
+Conversely, a sensor's page lists the automations that write data to it.
+
+.. _automation_cursor:
+
+Appendix: how the runner decides what is due
+--------------------------------------------
+
+This section describes the bookkeeping behind the catch-up behaviour above.
+You do not need it to use automations.
+
+The runner is a stateless command, executed once a minute by cron, so it needs a durable record of how far each automation has got.
+That record is one UTC timestamp per automation, its *cursor*: the scheduled time of the most recent run the automation has committed to.
+Runs at or before the cursor are never queued again.
+Before queueing any jobs, the runner advances the cursor to the run it is about to queue, and saves it.
+The cursor therefore records that a run was claimed, not that queueing or the task itself succeeded.
+
+Keeping a single moving timestamp, rather than a record per run, is what makes the behaviour above fall out: a runner that has been down catches up by moving the cursor straight to the latest due run, and two runners started in the same minute cannot queue the same run twice, because the cursor is advanced with a conditional update that only one of them can win.
+
+A new automation starts from its creation minute and does not replay runs from before it existed.
+Changing its cron expression or timezone, or reactivating it, restarts from the time of that change.
+Deactivated automations do not accumulate catch-up work.
+After upgrading an existing installation, runs scheduled before the upgrade are not replayed.
+
+Daylight-saving-time transitions follow wall-clock semantics.
+If the clock skips a scheduled local time in spring, that run happens once at the transition boundary.
+If a scheduled local time occurs twice in autumn, the first instance is the canonical run and the repeated instance is not queued again.

documentation/features/forecasting.rst

@@ -213,3 +213,11 @@ Usage:
    Create the annotations you want to use as regressors before running the forecast.
    For holidays, use ``flexmeasures add holidays``, which supports both ``workalendar``
    and ``holidays``. See :ref:`annotations` for details.
+
+.. _automating_forecasts:
+
+Automating forecasts
+--------------------
+
+Instead of asking for forecasts one at a time, you can set up an *automation*: a recurring task defined on an asset, which queues forecasting jobs on a cron schedule.
+See :ref:`automations`.

documentation/index.rst

@@ -175,6 +175,7 @@ In :ref:`getting_started`, we have some helpful tips how to dive into this docum
     features/scheduling
     features/forecasting
     features/reporting
+    features/automations
 
 .. toctree::
     :caption: Tutorials

flexmeasures/api/common/utils/api_utils.py

@@ -26,6 +26,7 @@
 )
 from flexmeasures.data.models.generic_assets import GenericAsset
 from flexmeasures.data.models.time_series import Sensor
+from flexmeasures.data.queries.generic_assets import asset_is_in_subtree
 from flexmeasures.data.utils import (
     SAVE_TO_DB_SUCCESS,
     SAVE_TO_DB_SUCCESS_BUT_NOTHING_NEW,
@@ -576,23 +577,6 @@ def _determine_copy_name(
     return f"{source_name} (Copy {max_index + 1})"
 
 
-def _asset_is_in_subtree(root_asset_id: int, candidate_asset_id: int) -> bool:
-    """Return True if candidate_asset_id is root or a descendant of root_asset_id."""
-    current_asset_id = candidate_asset_id
-    visited: set[int] = set()
-
-    while current_asset_id is not None and current_asset_id not in visited:
-        if current_asset_id == root_asset_id:
-            return True
-        visited.add(current_asset_id)
-        current_asset = db.session.get(GenericAsset, current_asset_id)
-        if current_asset is None:
-            return False
-        current_asset_id = current_asset.parent_asset_id
-
-    return False
-
-
 def copy_asset(
     asset: GenericAsset,
     account=None,
@@ -640,7 +624,7 @@ def copy_asset(
             target_account_id = int(account.id)
             target_parent_asset_id = int(parent_asset.id)
 
-        if target_parent_asset_id is not None and _asset_is_in_subtree(
+        if target_parent_asset_id is not None and asset_is_in_subtree(
             root_asset_id=asset.id,
             candidate_asset_id=target_parent_asset_id,
         ):