Skip to content

Feat/2393 durable automation runs - #2457

Open
BelhsanHmida wants to merge 8 commits into
mainfrom
feat/2393-durable-automation-runs
Open

Feat/2393 durable automation runs#2457
BelhsanHmida wants to merge 8 commits into
mainfrom
feat/2393-durable-automation-runs

Conversation

@BelhsanHmida

@BelhsanHmida BelhsanHmida commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Description

  • Gave every scheduled automation run a durable record, so a run whose queueing failed before any job existed is retryable instead of lost.
  • Added three tables: automation_run (one row per scheduled run, unique on (automation_id, scheduled_at, schedule_revision)), automation_run_job (the jobs a run intends to create, with deterministic RQ job IDs) and automation_run_attempt (one row per dispatch attempt).
  • Added Automation.schedule_revision, counted up whenever flexmeasures edit automation rebases the cursor, so runs of the old and the new schedule stay distinct.
  • Moved run ownership from the Redis SET NX guard to the database, keeping the same cursor compare-and-swap. Removes claim_due_automation() and the Redis guard.
  • Gave each claim a ten-minute lease, so a runner that died mid-queueing hands its work over, while one still queueing is left alone.
  • Persisted the run's plan before the first enqueue, so a retry re-queues only the jobs still missing, with the parameters the run was planned with.
  • Tracked dispatch state (pending, claimed, partially_queued, queued, failed) separately from execution state (pending, running, succeeded, failed, canceled), the latter derived from all job intents so a late success cannot hide an earlier failure.
  • Recorded job start, success and failure against the run, rolling the session back first so an outcome survives a job that failed on the database.
  • Added schedule_revision and a run_stats object to the automations API, and the same run status to the asset's Automations page. Backward compatible: no existing field changed.
  • Made flexmeasures jobs run-automations report the run and attempt it works on, and retry runs whose queueing did not finish.
  • Added changelog item in documentation/changelog.rst

Look & Feel

flexmeasures jobs run-automations, on a run whose previous attempt queued only part of its jobs:

Automation 42 ('Partial run') run 7 queued 3 forecasting job(s), scheduled for 2026-08-05 01:00:00+00:00.

and when an attempt fails, naming the run and the attempt rather than just the automation:

Automation 42 ('Partial run') run 7 failed while dispatching attempt 2: lost Redis connection

[GET] /assets/(id)/automations/(automation_id) gains run_stats:

"run_stats": {
  "total": 4,
  "dispatch": {"queued": 3, "partially_queued": 1},
  "execution": {"succeeded": 3, "pending": 1},
  "latest_run": {
    "id": 12,
    "scheduled_at": "2026-07-11T04:00:00+00:00",
    "schedule_revision": 1,
    "dispatch_state": "partially_queued",
    "execution_state": "pending",
    "attempt_count": 2,
    "intended_job_count": 2,
    "queued_job_count": 1,
    "claim_owner": "runner-1@host",
    "last_error": {"type": "ConnectionError", "message": "lost Redis connection"},
    "latest_attempt": {"attempt_no": 2, "owner": "runner-1@host", "outcome": "failed", "queued_job_count": 1},
    "jobs": [
      {"logical_job_key": "cycle-001", "rq_job_id": "automation-run-12-cycle-001", "status": "queued"},
      {"logical_job_key": "wrap-up", "rq_job_id": "automation-run-12-wrap-up", "status": "pending"}
    ]
  },
  "recent_runs": []
}

The automation details modal on the asset's Automations page shows the schedule revision and a Durable runs block with the same totals, dispatch and execution breakdown, and latest run — next to the existing Recently created jobs counts, which come from Redis and disappear as jobs expire, whereas the run records do not.

How to test

# 1. create an automation and let it become due
flexmeasures add automation --asset 3 --name "Daily PV forecasts" --cron "* * * * *" --sensor 12
flexmeasures jobs run-automations

# 2. inspect the durable run, on the asset's Automations page or through the API
curl -H "Authorization: $AUTH_TOKEN" \
  "$HOST/api/v3_0/assets/3/automations/<automation_id>" | jq .run_stats

# 3. break the dispatch halfway (stop Redis, or kill the runner mid-queueing), then run it again.
#    The retry queues only the jobs still missing, on the same run record, with attempt_count
#    incremented — no second run row, and no duplicate jobs.

Automated coverage:

pytest \
  flexmeasures/data/tests/test_automation_runs_fresh_db.py \
  flexmeasures/data/tests/test_automation_scheduling_fresh_db.py \
  flexmeasures/data/tests/test_automations_fresh_db.py \
  flexmeasures/cli/tests/test_automations.py \
  flexmeasures/api/v3_0/tests/test_automations_api.py \
  flexmeasures/api/v3_0/tests/test_automations_api_fresh_db.py

Related Items

Closes #2393.

Sign-off

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

@BelhsanHmida
feat(data/models): add durable automation run records
729218e 
An automation occurrence currently leaves no trace of its own: the only durable
marker is the automation's scheduling cursor, which says an occurrence was
attempted, not what happened to it.

Record each occurrence a runner picks up as an AutomationRun, with the attempts
made on it (AutomationRunAttempt) and the jobs it intends to create
(AutomationRunJob). Dispatch progress and worker execution outcome are tracked
separately, because 'queued everything' and 'the jobs succeeded' are different
questions an operator needs answered.

The database enforces one run per automation, occurrence and schedule revision,
and one job intent per run and logical job key. The new schedule revision on
Automation keeps runs of an edited or reactivated schedule apart from the runs
of the schedule it replaced, even at the same scheduled UTC time.

All run timestamps are validated to be timezone-aware and stored as UTC.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
@BelhsanHmida
feat(data/services): claim automation occurrences durably and resume …
30ef47b 
…partial dispatch

The runner used to advance and commit the scheduling cursor before queueing
anything, guarded only by a Redis key with a two-minute TTL. A failure before
the first enqueue therefore lost the occurrence for good, while retrying a
partial enqueue could have duplicated work.

Claim each due occurrence into an AutomationRun instead, write down the plan for
the run before the first enqueue, and give every intended job a logical key and,
from it, a deterministic RQ job ID. A retry replays that stored plan: jobs whose
IDs are already in Redis are recognised and left alone, and only the missing ones
are queued. Because the plan holds the parameters and timings the occurrence was
planned with, a retry hours later still dispatches the occurrence as originally
intended, even if the automation has been edited since.

Ownership is a database lease, not a Redis key. An occurrence is only picked up
by another runner once the lease of the runner holding it has expired, which is
how a runner that died mid-queueing hands its work over. A runner that fails
releases its own lease, so its run is retryable at once. Dispatch is finished
only when it is marked complete, so a crash between the last enqueue and that
mark is finalized by the next runner rather than left hanging.

Forecast cycle and wrap-up jobs now carry their run identity and report their
own start, success and failure back to it, so the execution outcome outlives the
Redis jobs. Queueing the jobs of a pipeline run moved out of the already long
run() into its own methods, which also removes the duplicate queueing path.

Editing an automation's cron string or timezone, or reactivating it, counts up
its schedule revision, so runs of the old and new schedule stay distinct.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
@BelhsanHmida
feat(api/ui): show durable automation run status
01e135a 
Job counts came from Redis alone, so once its jobs expired there was no way to
tell whether an occurrence had failed before queueing anything, queued only part
of its work, or queued everything and then failed while computing.

Add a run_stats object to the automation detail response, summarizing the
automation's durable runs and describing the recent ones: their occurrence,
dispatch and execution state, attempt count, intended and queued job counts,
timestamps, last error, latest attempt, and the jobs they created. Automation
responses also expose schedule_revision. The automation details panel shows the
latest run alongside the recent Redis jobs.

Both additions are additive: no existing field changed.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
@BelhsanHmida
docs: describe durable automation runs and safe retries
ea677df 
The forecasting docs and the CLI change log stated that a failed or partially
completed queueing attempt is never retried, which no longer holds. Describe how
an occurrence is claimed, planned, dispatched and retried instead, and what the
dispatch and execution states mean, along with the changelog entries for the new
run status in the API and the UI.

Those two stale CLI change log lines are corrected rather than left standing:
they describe an unreleased release, so leaving them would ship a change log
that contradicts itself.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
@BelhsanHmida
refactor(data/models): keep the dispatched job statuses in one place
b63a42b 
The tuple of statuses that count a job intent as dispatched was written out in
the model and, unused, in the service. Define it once next to the model that
asks the question, and drop the copy nobody read. Also apply black to the
migration and drop an unused import from the new test module.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
@BelhsanHmida
fix(data/services): keep recording a job outcome when the job broke t…
7a6fd5f 
…he transaction

Running an automation against a real database and a real Redis turned up two ways
the durable execution record lied about what happened.

A cycle job that failed on the database itself left the session in an aborted
transaction, so recording its failure was refused and the job stayed 'running'
forever. Roll back before recording, the way dispatch failures already do; the
job's uncommitted work is lost either way, since it is failing.

A later job succeeding also reset the run to 'running', burying an earlier
failure: the wrap-up job succeeds whatever became of the cycle jobs it reports
on. Derive the run's execution state from all of its jobs instead, so a failed
job keeps the run failed.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
@BelhsanHmida
docs/changelog: link the durable automation runs PR
825b206 
The entry linked the issue, because no PR existed when it was written.
Point it at PR #2457 instead, which is what the changelog convention asks for.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
@read-the-docs-community

read-the-docs-community Bot commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Documentation build overview

📚 flexmeasures | 🛠️ Build #34364441 | 📁 Comparing 78301ef against latest (d4e0805)

  🔍 Preview build  

11 files changed · ± 11 modified

± Modified

@BelhsanHmida
fix(api/tests): stop the sensor-data error listener from outliving it…
78301ef 
…s test

test_post_sensor_data_twice registers a 'handle_error' listener on the Engine class, which is global and process-wide, and never removes it. Every database error raised by any later test therefore ran its assertion that the error is an IntegrityError, so a test which provokes a different error fails inside SQLAlchemy rather than where it looks.

Remove the listener in a finally block, so it only covers the posts it is about.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add durable automation-run records and safe retry semantics

1 participant

@BelhsanHmida