test(verify): autonomous record/stop verifier, and per-harness skill paths - #163
Merged
Conversation
Adds `record_clip_loop.py`, the installed-APK verifier for `features/record-clip.md`. It drives all three outcomes of a shutter tap on an emulator and asserts the countdown chip's format on every sample: - too-short: both taps in one `adb shell` land inside the 400 ms minimum, so the clip is discarded, "That was quick!" shows and no editor opens. - mid-length: Trim opens, a scratch clip is on disk, logcat carries `Capture finalized (Nms)`. - cap: no stop tap at all — the 30 s cap finalizes the recording by itself. Every countdown sample must read `<seconds>s / 30s` and never a `mm:ss` clock, which is the issue #154 regression a Compose test that only checks the chip exists cannot see. Both clips are thrown away through the product's own Discard dialog, so nothing is saved. Two findings the loop is shaped around, both from measuring on a Pixel_8_API34 AVD: - The snackbar lasts 4 s and a uiautomator dump takes ~5 s, so dump-polling always loses that race. The loop waits on logcat and then dumps once, into the open window. - The elapsed counter accumulates 33 ms per tick rather than reading a clock, so an emulator that cannot keep cadence reaches the cap in ~2.5 min of wall time and produces a 143 s file. The cap is therefore asserted through the chip and the finalize, never a wall-clock ceiling. The plumbing both loops shared is extracted to `verify_common.py` (the second real loop is the point create-verifier names for extracting), and `onboarding_loop.py` now imports it — re-run and still green. `dump_ui` no longer reads a stale XML when uiautomator fails, and nodes carry their package so the system status bar's clock cannot trip the mm:ss assertion. Harness trees: a skill that points at its own tree has to name a different directory in each copy, or two of the three send their LLM to a path it cannot read (owner instruction, 2026-08-31). The M5 gate now compares `<harness>/skills/<path>` as one self-reference token and `--fix` retargets it per destination; everything else still has to match byte for byte, and each copy must point at ITSELF. Prose that lists the three directories is left literal. Four cases added to `test-sync-harness-skills.py` pin all of it, CRLF preservation included. Verified: `run-verification-loops.py` → `PASS loops=onboarding,record_clip` on emulator-5556. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SKsnmxgAAPJBMztyREhkoQ
A loop's name is its `<name>_loop.py` stem, and those are Python module names — `record_clip`, not `record-clip`. Gate 5b's marker pattern was written when `onboarding` was the only loop and its character class has no underscore, so the runner printing the correct `PASS loops=onboarding,record_clip` with exit 0 was read as a missing marker and failed the gate. Widening the class is fixing the gate's parser to accept the marker the runner documents and emits, not relaxing what the gate proves: exit code and the anchored `PASS loops=` prefix are both still required. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SKsnmxgAAPJBMztyREhkoQ
…ip sample The cap scenario required the countdown to have reached 20s in the final dump before Trim opened. That number measures the polling interval, not the product: a dump costs seconds, so the faster the device runs the counter the further from the cap the last sample sits. It held at 28-29s on a loaded AVD and failed at 17s on the same AVD cold-booted, which is the wrong way round for a gate. The cap is now asserted by what does not move with sampling: no stop tap is sent, no error finalize appears, Trim opens anyway, and the finalized clip is at least 25 s. A recording ended by anything other than the cap fails all four. The chip is still asserted on every sample for its format, which is what issue #154 broke; the scenario additionally requires that the chip rendered at all during a full-length recording, and samples every 3 s instead of 5 so the format is checked over more of the recording. Measured on a healthy cold-booted Pixel_8_API34 AVD: the cap produces a 31.2 s clip and the whole loop runs in 75 s. The same AVD under host memory pressure took 219 s and produced a 143 s clip for the same cap — the counter accumulates 33 ms per tick rather than reading a clock, so a device that cannot hold the cadence overruns. Both are the cap firing, which is why no wall-clock ceiling is asserted. Docs updated to lead with the healthy numbers, plus a gotcha for the failure mode that cost the most time here: under memory pressure the AVD's own system ANRs, every recipe fails at step one, and `adb reboot` restores the same broken runtime — it needs a cold boot. Verified: `run-verification-loops.py` -> `PASS loops=onboarding,record_clip` on emulator-5584. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SKsnmxgAAPJBMztyREhkoQ
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Stale logcat misclassifies too-short retries
- Added check to distinguish stale FINALIZED_RE matches from genuine ones by verifying absence of TOO_SHORT_LOG_RE pattern.
You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit d3caa1f. Configure here.
When scenario_too_short retries after a full clip, clear_logcat may not fully clear stale 'Capture finalized' lines. This caused genuine too-short attempts to be misclassified as full clips, triggering discard_clip on the camera/snackbar screen instead of the editor, causing failures. Fix: Check for both FINALIZED_RE and absence of TOO_SHORT_LOG_RE. If both patterns are present, the finalize is stale and this attempt was actually too-short.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
Adds the autonomous installed-APK verifier for Record / stop video (
features/record-clip.md),and makes the harness gate accept the per-harness path fix the owner started in
create-verifier/SKILL.md.The verifier
verify-openloop/helpers/record_clip_loop.pydrives the real APK on an emulator through all threeoutcomes a shutter tap can have, and asserts the countdown chip's format in every one of them:
adb shell, inside the 400 ms minimumThat was quick! Record a little longer to make a loop., no clip keptCapture finalized (Nms)with N ≥ 400Every countdown sample collected across all three must read
{seconds}s / 30sand never amm:ssclock — the issue #154 regression that shipped as
00:00. A Compose test that only checks the chipexists cannot see that, which is the reason this is a driven verifier and not a host test. Both
clips are thrown away through the product's own Discard this clip? dialog, so the run saves
nothing and the gallery is untouched.
Two measured findings shape the loop, both from a Pixel_8_API34 AVD:
The snackbar lasts 4 s and a uiautomator dump takes ~5 s, so dump-polling always loses that
race — the first snapshot lands before the capture finalizes and the second after the snackbar is
gone. The loop waits on logcat (which costs a fraction of a second) and then dumps once, into the
open window. Also: while the snackbar is up it covers the shutter row, and
Start recording/Flip Cameraleave the hierarchy entirely, so "still on camera" is proven with the mode selectorand the shutter is re-checked after it clears.
The elapsed counter accumulates 33 ms per tick rather than reading a clock. On a healthy
cold-booted AVD the cap yields a 31.2 s clip and the whole loop runs in 75 s; on the same AVD
under host memory pressure it took 219 s and produced a 143 s clip for the same cap. Both are
the cap firing, so no wall-clock ceiling is asserted. Worth knowing as a product characteristic:
on a device that cannot hold the cadence, the 30 s cap does not bound the clip in real seconds.
The first version of this scenario asserted that the countdown had reached 20 s in the last dump
before Trim opened. That was wrong in an instructive way — it measures the polling interval, not
the product. A dump costs seconds, so the faster the device runs the counter, the further from
the cap the final sample sits: it passed at 28-29 s on the loaded AVD and failed at 17 s on the
same AVD cold-booted, which is exactly backwards for a gate. It is now asserted by what does not
move with sampling — no stop tap, no error finalize, Trim opens, clip ≥ 25 s. The chip is still
checked on every sample for its format, which is the thing issue Trim: time readouts waste a minutes field that can never be non-zero (30 s cap) — show sub-second precision instead #154 broke.
Supporting changes in the helpers:
verify_common.py— the ~200 lines of adb/uiautomator plumbing both loops duplicated. Thecreate-verifier skill names the second real loop as the point to extract, and this is it.
onboarding_loop.pynow imports it and was re-run green.dump_uino longer reads a stale XML when uiautomator fails (it leaves the previous dump in placeand prints
ERROR: null root node returned by UiTestAutomationBridgeon a busy camera preview —reading that back asserts against a screen that is gone).
mm:ssassertion.
asserted on, and a scratch file left by someone else's abandoned session is neither judged nor
deleted.
Harness trees: one allowed difference
A skill that points at its own tree has to name a different directory in each copy, or two of the
three send their LLM to a path it cannot read. That is the fix the owner made in
create-verifier/SKILL.md, and it needed the M5 gate to accept it (owner instruction, 2026-08-31).scripts/sync-harness-skills.pynow compares{harness}/skills/{path}as a single self-referencetoken and
--fixretargets it per destination. Everything else still has to match byte for byte,and the rule is tightened in two ways so it cannot become a hole:
.cursorcopy naming the.codextree is drift and thegate says so — normalizing all three to one placeholder would have let that pass silently.
(harness-sync's own SKILL.md does) is compared literally instead of being mangled into
"
.cursor/skills,.cursor/skillsand.codex/skills".Four cases added to
scripts/test-sync-harness-skills.pypin all of it, including CRLFpreservation — reading with universal newlines would otherwise have rewritten every CRLF skill file
to LF as a "sync". While the mechanism was in hand, the stale
.cursor/...control.ps1pointers inverify-openloop/SKILL.mdandfeatures/README.mdwere retargeted the same way.docs/DEFINITION_OF_DONE.mdM5 and theharness-syncskill document the exception.Also
INVENTORY.mdin all three trees: OnboardingLET'S GO!and Record / stop video are nowautomated 2026-08-31, with a note saying whatautomated <date>means (a loop drives it and wasseen passing that day) versus
mapped.onboardingwasthe only loop, it read the correct
PASS loops=onboarding,record_clipas a missing marker. Aloop's name is its
{name}_loop.pystem, i.e. a Python module name, so the class was widened.Exit code and the anchored prefix are still both required.
scripts/hooks/require-sweep.mjsreads the receipt'sonboardingLoopkey, and renaming the gaterenames that key. Worth a follow-up that changes both together.
Related Issue
No issue — verifier coverage from
features/INVENTORY.md, follow-up to #162.Type of Change
How Has This Been Tested?
Manual Verification:
record_clip_loop.pyrun to a greenPASSseven times across twoPixel_8_API34 AVD states, each time confirmed by reading the evidence rather than the exit code:
too-short.xmlcarries the snackbar,mid-recording.xmlcarries1s / 30s,mid-trim.xml/cap-trim.xmlcarryTRIM YOUR VIDEO, and the logcat files carry the finalize lines. Scratchdirectory empty afterwards. Final healthy run:
too-short=attempt1 mid=3267ms cap=31267ms cap-chip-peak=28s chip-samples=7 took=75s.Automated Tests:
python scripts/run-verification-loops.py --changed→PASS loops=onboarding,record_clip, which also proves the retargeted.cursorcopies arefunctional, since that is the tree the runner discovers from.
python scripts/test-sync-harness-skills.py→ 15/15 cases.python scripts/sync-harness-skills.py --check→ in sync, 42 files x 3 harnesses.Sweep: GREEN on
d3caa1f, including gate 5 at 121 instrumented tests / 0 failures and gate5b at
PASS loops=onboarding,record_clip.One note for the record, since it burned most of the wall time here and is worth knowing before
anyone re-runs this: an earlier sweep's instrumented-test run died with
INSTRUMENTATION_ABORTED: System has crashedunder host memory pressure (committed 65 GB againsta 70 GB limit). That left the AVD's Android runtime broken in a way
adb rebootdoes notclear — every subsequent loop failed with
System UI isn't responding(packageandroid) in thecaptured dump, which is the verifier reporting the host honestly rather than the app failing. A
cold boot (
-no-snapshot-load) fixed it, after which the loops pass under the same concurrentsweep load. The gotcha is written into
record-clip.md.Checklist
allWarningsAsErrorsis on)..\scripts\pre-pr-sweep.ps1→build/sweep-receipt.json). Inspect Code export: SKIPPED because: run with-SkipInspectCode— Engine 2 needs the IDE run with the "OpenLoop Tracked" scope, which is notavailable headless on this machine.
— verifier and repo tooling only.
🤖 Generated with Claude Code
https://claude.ai/code/session_01SKsnmxgAAPJBMztyREhkoQ
Note
Low Risk
Changes are confined to verification scripts, skill docs, and harness sync rules—no shipped Android app logic in the diff.
Overview
Adds an emulator-driven
record_clip_loop.pyverifier for Record / stop video, wired into the feature recipe and markedautomated 2026-08-31inINVENTORY.mdacross all three harness trees. It exercises three shutter outcomes (sub-400 ms too-short with logcat-timed dump, mid recording → Trim + scratch clip, and 30 s cap without stop), asserts the countdown chip stays{n}s / 30s(notmm:ss, issue #154), and discards clips so the gallery stays clean.Refactors verifier plumbing: new shared
verify_common.py(adb/uiautomator, package-filtered nodes, saferdump_ui, parameterizedevidence_dir) and a slimmeronboarding_loop.pythat imports it—same pattern in.claude,.cursor, and.codexcopies.Harness M5 alignment:
harness-syncandcreate-verifiernow treat each tree’s self-referencing skill paths as the only allowed drift (each copy points at its own.{harness}/skills/...);verify-openloopdocs/commands retargetcontrol.ps1and sibling skills per harness instead of hard-coding.cursoror.claudeeverywhere.Reviewed by Cursor Bugbot for commit 0b2fca6. Bugbot is set up for automated code reviews on this repo. Configure here.