Skip to content

feat(runs): report a live phase for every automation run - #353

Open
aartem1 wants to merge 8 commits into
OpenHands:mainfrom
aartem1:feat/automation-run-phases
Open

feat(runs): report a live phase for every automation run#353
aartem1 wants to merge 8 commits into
OpenHands:mainfrom
aartem1:feat/automation-run-phases

Conversation

@aartem1

@aartem1 aartem1 commented Aug 20, 2026

Copy link
Copy Markdown

HUMAN:

I checked this by hand on a local stack. First the cases in the screenshots on
the frontend PR (OpenHands/OpenHands#16740) — the phase on the automations
list, on the activity-log rows, and the full label on hover — all driven by
this service. On top of that I ran a real but mocked scenario: a mock
automation whose entrypoint does no real work but reports its phases through
the new endpoint. I watched the phases arrive in order as the run progressed,
and saw a failed run keep the phase it stopped at.

AGENT:

Verified beyond unit tests: pre-commit and the full pytest suite, plus the
end-to-end local dispatch described under "Testing" below.


Why

An automation run reports one thing for its entire lifetime: RUNNING. A run
that takes minutes looks identical at second 5 and second 300, and a run that
fails tells you that it failed, never where it stopped — provisioning, the
bundle upload, the entrypoint, or somewhere inside the automation's own code.

This adds a current phase to a run: a short machine code, a human label, and
the moment it was written. The service records its own preparation milestones;
code running inside the sandbox reports the rest through a new endpoint.

Relates to OpenHands/OpenHands#16572. Frontend side: OpenHands/OpenHands#16740

What changed

  • Storage — three nullable columns on automation_runs (phase_code,
    phase_label, phase_updated_at) plus migration 016. Nullable with no
    backfill: an existing run simply has no phase, which is the truth.
  • EndpointPOST /v1/runs/{run_id}/phase, authenticated like the
    existing completion callback, 404 for an unknown run and 403 for someone
    else's automation. Every write replaces the whole (code, label) pair —
    last write wins, only the current phase is kept, so there is no history
    table to retain or prune. At least one of code/label is required.
  • Service phases — the dispatcher records queued,
    sandbox_provisioning, bundle_upload and entrypoint_start as it
    prepares a run. A failed phase write never fails the run: a phase is
    telemetry, not control flow.
  • Preset phases — the prompt and plugin templates report preparing
    and running_agent from inside the sandbox, using the same endpoint any
    custom automation can use.
  • Docsdocs/run-phase-reporting.md is the recipe for custom
    entrypoints, including which credential to use.

The credential subtlety worth a reviewer's attention

The sandbox holds two keys. SESSION_API_KEY authenticates to the agent
server
; the automation service rejects it. Phase reports must use
AUTOMATION_CALLBACK_API_KEY (local mode) or OPENHANDS_API_KEY (cloud).
Getting this wrong is silent — the POST 401s, the caller swallows it because a
phase must never break a run, and phases simply never appear while every test
stays green. The preset templates and the docs both spell this out.

Testing

  • uv run pre-commit run --all-files — clean (yamlfmt, ruff format, ruff
    lint, pycodestyle, pyright).
  • uv run pytest tests/ — 1337 passed. 36 of those are new: the endpoint's
    contract (auth, ownership, validation, replacement semantics), the
    dispatcher milestones against a real DB seam, and AST assertions that both
    preset templates emit the phases with the right credential.
  • End-to-end on a local stack, not only unit tests: a custom automation
    whose entrypoint is a shell script that does no real work but reports real
    phases through the endpoint. Dispatching it walks
    queued → sandbox_provisioning → bundle_upload → entrypoint_start → preparing → … and finishes COMPLETED in ~56s; a second variant that
    reports FAILED through the completion callback keeps its last phase
    visible on the failed run, which is the point of the feature.

aartem1 and others added 3 commits August 20, 2026 12:46
Adds phase_code/phase_label/phase_updated_at to AutomationRun and a
POST /v1/runs/{run_id}/phase endpoint so code running inside the sandbox
can report what it is doing. The endpoint mirrors complete_run: same
credentials, same automation ownership check.

A phase is one value -- the (code, label) pair -- and every successful
write replaces it whole, so a field omitted from the request is stored as
NULL rather than keeping its previous value. Merging would let a stale
code sit beside a fresh label as a pair nobody ever wrote.

Validation rejects with 422 before any write, leaving run status and the
stored phase untouched: label at most 200 characters, at least one of
code/label non-blank, and no Unicode Cc/Zl/Zp characters in either field.
Cf is deliberately not rejected -- it would ban the zero-width joiners
inside emoji sequences and the bidi marks non-Latin labels carry.

Segment S01 of tasks/IMPL-automation-run-phases.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Between accepting a run and starting its entrypoint the service was a
blind spot: the run sat in RUNNING while a sandbox was provisioned and a
bundle uploaded, and nothing told a hung run from a working one. The
dispatcher and the shared execution path now record four milestones --
queued, sandbox_provisioning, bundle_upload, entrypoint_start -- which
the frontend translates.

They are written through _record_run_phase, which writes the columns
directly rather than calling the service's own REST endpoint, and which
never raises: a phase is telemetry, so failing to record one must not
fail the run. Writing stops at entrypoint_start so the service cannot
stamp over a phase the automation itself reports afterwards.

Both call sites sit in code shared by the cloud and local backends, so
the milestones exist in sandboxless mode too.

Also injects AUTOMATION_PHASE_URL beside AUTOMATION_CALLBACK_URL.
The callback variable turned out to be the full /complete URL rather
than a base, so automation code had no address to build the phase
endpoint from, and chopping the suffix off would be brittle.

Segment S02 of tasks/IMPL-automation-run-phases.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Once the entrypoint starts, the service stops writing phases and the run
goes dark for the rest of its life -- which is where the interesting work
happens. Both presets now report "preparing" and "running_agent" through
POST /v1/runs/{run_id}/phase, and docs/run-phase-reporting.md gives the
author of a custom automation the same recipe.

Reporting is best-effort: a missing AUTOMATION_PHASE_URL, a network
error, a timeout or a non-2xx response are all swallowed, and the request
carries a timeout so a hung endpoint cannot stall the automation.

The credential is AUTOMATION_CALLBACK_API_KEY in local mode and
OPENHANDS_API_KEY in Cloud -- the two the backends actually inject.
SESSION_API_KEY is deliberately not used despite being present in the
sandbox: it authenticates to the agent server, a different service, and
the local automation service accepts only its own key, so sending it
would 401 and drop every phase silently.

Preset templates are never executed by the suite, so the tests assert on
the module AST instead of source text: every call site's literal
arguments are fed to the real RunPhaseRequest, each call is checked to be
unnested, and the urllib import is checked to exist -- a deleted import
would otherwise raise inside the helper's own try and be swallowed.

Segment S03 of tasks/IMPL-automation-run-phases.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@aartem1
aartem1 force-pushed the feat/automation-run-phases branch from 60a234a to 876a737 Compare August 20, 2026 09:49
A phase write was accepted whatever the run's status, so a sandbox that
outlived its run — or anything else holding the same credential — could
move the phase of a run that had already finished. For a failed run that
phase is the record of where it stopped, and the UI shows it beside the
failure, so overwriting it destroys the only account of how far the run
got.

Only PENDING and RUNNING runs now accept a phase; anything else answers
409, the same way `cancel_run` refuses a terminal run. The status is
re-checked in the UPDATE itself, because the completion callback can land
between the check and the write.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aartem1 and others added 3 commits August 21, 2026 10:00
The contract lets a phase carry only a code, and the UI shows that code
raw when it does — worth knowing before shipping `checking_out` to a
user's screen.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The label is stored as sent — only a phase blank on both fields is
rejected — so the UI trims it and falls back to the code rather than
rendering blank text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@all-hands-bot

Copy link
Copy Markdown
Contributor

👋 This PR needs a couple of things fixed before OpenHands can review it:

  • the PR description's HUMAN: section needs at least 20 characters describing what you tested, not just the template placeholder

Push an update once this is addressed and this check re-runs automatically.

This is an automated check - no AI was used to generate this comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants