MobBridge template: screenshot + id-addressable scroll + element frames#19
Merged
Conversation
…arness
Add the Kotlin side of mob's in-process test-harness driving (paired with
mob's screenshot/3, scroll_info/1, scroll_to/3 NIFs):
- screenshot(format, quality, scale): PixelCopy of the activity window →
PNG/JPEG bytes (decor-view draw fallback pre-API-26).
- An id-keyed ScrollHandle registry (ScrollState + LazyListState +
measured viewport); the :scroll and lazy-list composables register
themselves by their :id prop and report viewport size via
onGloballyPositioned.
- scrollInfo(id) → flat JSON {offset,content,viewport,max,kind}
(kind "pixel" for verticalScroll, "index" for LazyColumn);
scrollTo(id, x, y) drives the registered scroll state on the main thread.
Lets a remotely-connected agent capture pixels and scroll deterministically
over Erlang dist with no adb. Verified on a moto g power (Android 11).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kotlin side of mob's element_frames NIF: RenderNodeInner attaches a frameTrackingModifier (Modifier.testTag(id) + onGloballyPositioned) to any node carrying an :id, recording its window bounds into elementFramesById. elementFrames() returns them as JSON in dp (matching screenInfo / tap_xy units); cleared on navigation. Lets a remotely-connected agent read element positions by id over dist with no screenshot. Verified on a moto g power. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GenericJam
added a commit
that referenced
this pull request
May 29, 2026
Kotlin side of mob's test harness (companion to mob 0.6.23): generated apps gain screenshot (PixelCopy), id-addressable scroll, and element frames so a remote agent can capture/scroll/locate over dist with no adb (#19). Also wires Android WebView onShowFileChooser for HTML file inputs. Fix: the new scroll/element-frame registries are read from the NIF thread while Compose writes them on the main thread; switch scrollHandlesById and elementFramesById from mutableMapOf to ConcurrentHashMap (computeIfAbsent for handle creation) to remove the ConcurrentModificationException risk. iOS already guarded its registry with @synchronized. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kotlin side of mob's in-process test-harness driving (companion to GenericJam/mob#40). Generated apps gain the
MobBridgemethods the mob Zig NIFs call so a remotely-connected agent can capture, scroll, and locate elements over Erlang dist with noadb.Added to
MobBridge.kt.eexscreenshot(format, quality, scale)—PixelCopyof the activity window → PNG/JPEG bytes (decor-viewdrawfallback pre-API-26).ScrollHandleregistry (ScrollState + LazyListState + measured viewport);:scroll/lazy-list composables register by:idand report viewport viaonGloballyPositioned.scrollInfo(id)→ flat JSON{offset,content,viewport,max,kind}(pixelvsindex);scrollTo(id, x, y)drives on the main thread.frameTrackingModifier(id)(testTag+onGloballyPositioned) attached inRenderNodeInnerfor any node with an:id→elementFramesById;elementFrames()→ JSON{id:[x,y,w,h]}in dp. Cleared on navigation.Opt-in per
:id; untagged nodes get no tracking modifier.Must land with
GenericJam/mob#40 (the NIFs that call these methods). Verified end-to-end on a moto g power (Android 11) via a generated app.
🤖 Generated with Claude Code