Skip to content

feat: act on the app's own controls in-process (elementActionRequest) - #4

Merged
sarensw merged 1 commit into
mainfrom
sarensw/element-actions
Aug 15, 2026
Merged

feat: act on the app's own controls in-process (elementActionRequest)#4
sarensw merged 1 commit into
mainfrom
sarensw/element-actions

Conversation

@sarensw

@sarensw sarensw commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Why

Launch parameters put an app into a state. Some screenshots also need its UI driven — MacPacker's search field has to be focused for the search shot, which until now the app did itself on seeing -SearchQuery, i.e. screenshot-shaped behaviour living in shipping window code.

Driving it from outside would mean synthesizing input, which needs the Accessibility grant. Driving it inside the target does not: a process may read and drive its own accessibility tree freely. That is the same trick ElementLocator.screenFrame already uses for element frames — this just adds acting to looking.

What

elementActionRequest names a control by accessibility identifier and says what to do to it. ElementLocator.perform resolves it with the same in-process self-read and replies elementActionResult — whether it worked, and if not why, so a plan naming a control the app doesn't expose fails legibly instead of producing a screenshot of the wrong thing.

public struct ElementAction: Codable, Sendable {
    public enum Kind: String, Codable, Sendable { case focus }
    public let identifier: String
    public let kind: Kind
}

One kind, focus. Add another when a plan needs one.

Neither side learns anything about any app's UI: the identifier comes from whoever wrote the plan. The app's only obligation is ordinary accessibility — .accessibilityIdentifier(_:) on the control.

Verified

swift test: 10 tests in 3 suites, all passing — both new message shapes round-trip, and acting on an unknown identifier reports why rather than silently succeeding.

Consumers build clean against this branch (SandboxPilot and MacPacker).

Not yet verified end to end: that a real focus reaches a real control. This machine has no Accessibility/Automation grant for the harnesses that could check it independently — the XCUITest runner fails with "Timed out while enabling automation mode" — so it needs a run of MacPacker's 04-search scenario through a SandboxPilot built from this branch. Worth doing before tagging.

Needs a 1.4.0 tag after merge; LeanBytes/SandboxPilot#32 and sarensw/MacPacker#182 both pin it.

Where launch parameters put an app into a state, a plan sometimes needs to
drive its UI — MacPacker's search field has to be focused for the search
screenshot. Doing that from outside would mean synthesizing input, which
needs the Accessibility grant; doing it inside the target does not, because
a process may read and drive its own accessibility tree freely.

elementActionRequest names a control by accessibility identifier and says
what to do to it. ElementLocator resolves it with the same in-process self
read as screenFrame, and replies with elementActionResult saying whether it
worked and, if not, why — so a plan naming a control the app does not expose
fails legibly instead of producing a screenshot of the wrong thing.

One action kind, focus. Neither the kit nor SandboxPilot learns anything
about any app's UI: the identifier comes from whoever wrote the plan.
@sarensw
sarensw merged commit 5733ab7 into main Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant