Run reports via CLI and API

[Flix6x]

Description

Closes #2426.

This PR makes one-off reports available as background jobs, independently from report automations. It brings the non-automation-specific reporting infrastructure forward from #2297, #2290 and #2294 so the CLI and API functionality can be reviewed, released and tested on its own.

• Register a dedicated reporting RQ queue with configurable job timeouts and worker support.
• Add flexmeasures add report --as-job for queueing reports from the CLI.
• Add POST /api/v3_0/assets/<id>/reports/trigger for queueing reports through the API.
• Persist the reporter configuration before queueing so a worker can reconstruct the reporter.
• Store report results as beliefs and expose the job through the existing job-status and asset-status functionality.
• Resolve reporter input and output sensors through the data generator itself, including configuration-based dependencies such as Profit-or-loss price sensors.
• Require read access to every input and configuration sensor and create-children access to every output sensor.
• Require every output sensor to belong to the asset in the URL or one of its descendants.
• Validate and authorize the request before persisting the reporter data source or queueing work.
• Apply the shared schedule/forecast/report trigger rate limit.
• Add CLI, API, worker, authorization and end-to-end regression tests.
• Document the reporting worker queue, API endpoint and permission model.
• Added changelog item in documentation/changelog.rst.

The API request envelope is validated first. The selected reporter then validates config and parameters with its concrete schemas. FlexMeasures resolves and authorizes all declared dependencies before creating the reporting job.

Look & Feel

CLI

Queue a report instead of computing it in the CLI process:

flexmeasures add report \
  --reporter PandasReporter \
  --config report-config.json \
  --parameters report-parameters.json \
  --start 2023-04-10T00:00:00+00:00 \
  --end 2023-04-10T10:00:00+00:00 \
  --as-job

Process queued reports with a reporting worker:

flexmeasures jobs run-worker --queue reporting

--as-job stores report results in the database and therefore cannot be combined with --dry-run or --output-file.

API

The endpoint is included in the generated OpenAPI specification and can be tried interactively through Swagger UI:

1. Start FlexMeasures and sign in with a user who can access the target asset.
2. Open http://127.0.0.1:5000/api/v3_0/docs.
3. Under Assets, expand POST /api/v3_0/assets/{id}/reports/trigger.
4. Select Try it out.
5. Enter the target asset ID, for example 42.
6. Paste the following request body, replacing the sensor IDs with sensors available to your user.

{
  "reporter": "PandasReporter",
  "config": {
    "required_input": [
      {"name": "one", "unit": "kW"},
      {"name": "two", "unit": "kW"}
    ],
    "required_output": [
      {"name": "sum", "unit": "kW"}
    ],
    "transformations": [
      {
        "df_input": "one",
        "method": "add",
        "args": ["@two"],
        "df_output": "sum"
      }
    ]
  },
  "parameters": {
    "input": [
      {"name": "one", "sensor": 12},
      {"name": "two", "sensor": 13}
    ],
    "output": [
      {"name": "sum", "sensor": 14}
    ],
    "start": "2023-04-10T00:00:00+00:00",
    "end": "2023-04-10T10:00:00+00:00"
  }
}

Select Execute. Swagger UI shows the generated request and the server response. A valid request returns HTTP 202 Accepted using the canonical asynchronous-job response:

{
  "status": "ACCEPTED",
  "message": "Request has been accepted for processing.",
  "job": "<job UUID>",
  "job-url": "/api/v3_0/jobs/<job UUID>"
}

The response is defined by the generated OpenAPI documentation. The caller can follow job-url or use GET /api/v3_0/jobs/{uuid} in Swagger UI until the job finishes. The queued job records {"origin": "API"} in its metadata.

How to test

Automated tests

python -m pytest -q \
  flexmeasures/api/v3_0/tests/test_report_trigger_api.py \
  flexmeasures/api/v3_0/tests/test_rate_limiting.py \
  flexmeasures/cli/tests/test_data_add.py \
  flexmeasures/data/schemas/tests/test_reporting.py \
  flexmeasures/data/models/reporting/tests \
  flexmeasures/data/tests/test_generic_assets.py \
  flexmeasures/data/tests/test_reporting_service.py \
  flexmeasures/utils/tests/test_job_utils.py

The focused suite covers:

• CLI queueing and reporter reconstruction from the stored data source.
• The canonical HTTP 202 response and job-status URL.
• API-to-worker execution and persisted output beliefs.
• Anonymous, cross-organisation and sensor-level permission failures.
• Source-filtered inputs and configuration-based price-sensor dependencies.
• Output subtree enforcement.
• Missing or invalid reporter inputs and outputs without database or queue side effects.
• Shared trigger rate-limit registration.
• Reporting queue configuration and timeout handling.

Latest local result: 119 passed, 22 warnings.

Manual test

1. Choose two input sensors containing historical data and an output sensor below the asset used in the API URL.
2. Create a valid Pandas reporter configuration and parameters file using fixed historical start and end times.
3. Run flexmeasures add report ... --as-job and confirm a job is added to the reporting queue.
4. Start flexmeasures jobs run-worker --queue reporting and confirm the job finishes and beliefs are stored on the output sensor.
5. Submit the equivalent API request and confirm it returns HTTP 202 with job and job-url.
6. Poll GET /api/v3_0/jobs/<job UUID> and confirm the job moves from queued to finished.
7. Repeat with an unreadable input/configuration sensor and confirm HTTP 403 with no reporter data source or job created.
8. Repeat with an output outside the URL asset subtree and confirm HTTP 422 with no reporter data source or job created.
9. Repeat without required inputs or outputs and confirm HTTP 422 with no persistence or queue side effects.

Further Improvements

Recurring report automations, automation UI work and prepared report templates remain outside this standalone PR and continue in their respective follow-up work.

Related Items

• Closes Run one-off reports as background jobs via CLI and API #2426
• Related to the broader automations story Automations - first roundtrip for forecasts #2288
• Extracted from the reports-as-automations work in Reports as jobs and as automations #2297

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or another incompatible license.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #34344007 | 📁 Comparing 4db23e1 against latest (4a5892f)

  🔍 Preview build  

145 files changed · + 26 added · ± 119 modified

+ Added

• _autodoc/COUPLING.html
• _autodoc/COUPLING_COEFFICIENT.html
• _autodoc/GROUP.html
• _autodoc/INFLEXIBLE_CONSUMPTION.html
• _autodoc/INFLEXIBLE_PRODUCTION.html
• _autodoc/OPERATION_MODES.html
• _autosummary/flexmeasures.api.common.rate_limiting.html
• _autosummary/flexmeasures.api.v3_0.utils.html
• _autosummary/flexmeasures.data.models.automations.html
• _autosummary/flexmeasures.data.models.planning.devices.html
• and 16 more...

± Modified

• changelog.html
• configuration.html
• genindex.html
• getting-started.html
• http-routingtable.html
• index.html
• py-modindex.html
• _autodoc/COMMITMENTS.html
• _autodoc/CONSUMPTION_CAPACITY.html
• _autodoc/INFLEXIBLE_DEVICE_SENSORS.html
• and 109 more...

[BelhsanHmida]

@nhoening Thanks, I followed your proposal and made #2298 standalone on main.

It now includes the reporting queue and worker, flexmeasures add report --as-job, the shared
data-generator sensor-resolution and authorization helpers, reporter input/output
dependencies, output-subtree enforcement, rate-limit documentation/tests, and the
API-to-worker end-to-end test that verifies stored beliefs.

I also added defensive validation for non-empty reporter inputs and outputs before
the reporter data source is persisted.

The ReportTriggerSchema docstring has been updated to clarify that it validates
only the request envelope, while the selected reporter validates its concrete
configuration and parameters.

I have also updated the PR description to reflect the 202 job/job-url response
and the enforced permission model. Ready for another review once CI finishes.

[nhoening]

This is looking pretty good. I have comments, but only small ones.

I did try the manual testing via SwaggerUI. It went okay, only I believe we are not failing when we should when computing the job.

I used the PandasReporter example from the PR description, but my input sensors have different units (EUR and kW, outut sensor has kW). I expected an error on this.

Instead, the worker shows no error, and the job result (via the job status endpoint) looks like this:

{
  "ended-at": "2026-09-01T16:33:56.598447+00:00",
  "ended_at": "2026-09-01T16:33:56.598447+00:00",
  "enqueued-at": "2026-09-01T16:32:45.590679+00:00",
  "enqueued_at": "2026-09-01T16:32:45.590679+00:00",
  "exc-info": null,
  "exc_info": null,
  "func-name": "flexmeasures.data.services.reporting.run_report_job",
  "func_name": "flexmeasures.data.services.reporting.run_report_job",
  "message": "Reporting job has finished.",
  "origin": "reporting",
  "result": [
    {
      "n_rows": 194,
      "sensor_id": 49348
    }
  ],
  "started-at": "2026-09-01T16:33:56.339937+00:00",
  "started_at": "2026-09-01T16:33:56.339937+00:00",
  "status": "FINISHED"
}

I checked and sensor 49348 has no data.

I am not sure if what I saw is something that belongs in this PR as a fix or not, but it would be good to investigate.

[nhoening]

Why is a commit needed here? I can't what changrd on reporter, as data_source was on it before already.

If we need to commit, why also do a flush right before?

[BelhsanHmida]

Added a comment for the first one.

The commit is needed because reporter.data_source is a lazy property (data_sources.py:245) that can create a brand-new source row. The worker is a separate process and our views don't auto-commit, so without it the worker's lookup by id fails. No test catches this work_on_rq runs SimpleWorker in-process on the same session which is why the comment seemed worth adding.

On the flush: you're right, it's redundant given the commit right after. It only matters because the id is read before the commit; moving that read below would let it go. Left it for consistency with the same block in train_predict.py:454-458 happy to drop it in both in a follow-up commit.

[BelhsanHmida]

This is looking pretty good. I have comments, but only small ones.

I did try the manual testing via SwaggerUI. It went okay, only I believe we are not failing when we should when computing the job.

I used the PandasReporter example from the PR description, but my input sensors have different units (EUR and kW, outut sensor has kW). I expected an error on this.

Instead, the worker shows no error, and the job result (via the job status endpoint) looks like this:

{
  "ended-at": "2026-09-01T16:33:56.598447+00:00",
  "ended_at": "2026-09-01T16:33:56.598447+00:00",
  "enqueued-at": "2026-09-01T16:32:45.590679+00:00",
  "enqueued_at": "2026-09-01T16:32:45.590679+00:00",
  "exc-info": null,
  "exc_info": null,
  "func-name": "flexmeasures.data.services.reporting.run_report_job",
  "func_name": "flexmeasures.data.services.reporting.run_report_job",
  "message": "Reporting job has finished.",
  "origin": "reporting",
  "result": [
    {
      "n_rows": 194,
      "sensor_id": 49348
    }
  ],
  "started-at": "2026-09-01T16:33:56.339937+00:00",
  "started_at": "2026-09-01T16:33:56.339937+00:00",
  "status": "FINISHED"
}

I checked and sensor 49348 has no data.

I am not sure if what I saw is something that belongs in this PR as a fix or not, but it would be good to investigate.

I verified this locally. The example did not declare expected units, so PandasReporter did not attempt unit conversion. The inputs also did not align on source and belief time, producing all-NaN rows that were dropped before persistence while the job incorrectly reported 194 rows.

I updated the example to declare kW units, added coverage confirming that a EUR input makes the job fail, and fixed the job result to report zero persistable rows instead of counting dropped NaN rows.