Skip to content

test(verify): autonomous record/stop verifier, and per-harness skill paths - #163

Merged
stozo04 merged 4 commits into
mainfrom
feature/record-clip-verifier
Sep 1, 2026
Merged

test(verify): autonomous record/stop verifier, and per-harness skill paths#163
stozo04 merged 4 commits into
mainfrom
feature/record-clip-verifier

Conversation

@stozo04

@stozo04 stozo04 commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Description

Adds the autonomous installed-APK verifier for Record / stop video (features/record-clip.md),
and makes the harness gate accept the per-harness path fix the owner started in
create-verifier/SKILL.md.

The verifier

verify-openloop/helpers/record_clip_loop.py drives the real APK on an emulator through all three
outcomes a shutter tap can have, and asserts the countdown chip's format in every one of them:

Scenario What it drives What it proves
too-short both taps in ONE adb shell, inside the 400 ms minimum stays on camera, That was quick! Record a little longer to make a loop., no clip kept
mid-length record a few seconds, tap stop Trim opens, a scratch clip is on disk, logcat has Capture finalized (Nms) with N ≥ 400
cap start and never tap stop the 30 s cap finalizes the recording by itself: no stop tap, no error finalize, Trim opens, clip is at least 25 s

Every countdown sample collected across all three must read {seconds}s / 30s and never a mm:ss
clock — the issue #154 regression that shipped as 00:00. A Compose test that only checks the chip
exists cannot see that, which is the reason this is a driven verifier and not a host test. Both
clips are thrown away through the product's own Discard this clip? dialog, so the run saves
nothing and the gallery is untouched.

Two measured findings shape the loop, both from a Pixel_8_API34 AVD:

  • The snackbar lasts 4 s and a uiautomator dump takes ~5 s, so dump-polling always loses that
    race — the first snapshot lands before the capture finalizes and the second after the snackbar is
    gone. The loop waits on logcat (which costs a fraction of a second) and then dumps once, into the
    open window. Also: while the snackbar is up it covers the shutter row, and Start recording /
    Flip Camera leave the hierarchy entirely, so "still on camera" is proven with the mode selector
    and the shutter is re-checked after it clears.

  • The elapsed counter accumulates 33 ms per tick rather than reading a clock. On a healthy
    cold-booted AVD the cap yields a 31.2 s clip and the whole loop runs in 75 s; on the same AVD
    under host memory pressure it took 219 s and produced a 143 s clip for the same cap. Both are
    the cap firing, so no wall-clock ceiling is asserted. Worth knowing as a product characteristic:
    on a device that cannot hold the cadence, the 30 s cap does not bound the clip in real seconds.

    The first version of this scenario asserted that the countdown had reached 20 s in the last dump
    before Trim opened. That was wrong in an instructive way — it measures the polling interval, not
    the product. A dump costs seconds, so the faster the device runs the counter, the further from
    the cap the final sample sits: it passed at 28-29 s on the loaded AVD and failed at 17 s on the
    same AVD cold-booted, which is exactly backwards for a gate. It is now asserted by what does not
    move with sampling — no stop tap, no error finalize, Trim opens, clip ≥ 25 s. The chip is still
    checked on every sample for its format, which is the thing issue Trim: time readouts waste a minutes field that can never be non-zero (30 s cap) — show sub-second precision instead #154 broke.

Supporting changes in the helpers:

  • verify_common.py — the ~200 lines of adb/uiautomator plumbing both loops duplicated. The
    create-verifier skill names the second real loop as the point to extract, and this is it.
    onboarding_loop.py now imports it and was re-run green.
  • dump_ui no longer reads a stale XML when uiautomator fails (it leaves the previous dump in place
    and prints ERROR: null root node returned by UiTestAutomationBridge on a busy camera preview —
    reading that back asserts against a screen that is gone).
  • Nodes carry their owning package, so the system status bar's own clock cannot trip the mm:ss
    assertion.
  • Scratch clips are diffed against a baseline taken at startup: only clips THIS run created are
    asserted on, and a scratch file left by someone else's abandoned session is neither judged nor
    deleted.

Harness trees: one allowed difference

A skill that points at its own tree has to name a different directory in each copy, or two of the
three send their LLM to a path it cannot read. That is the fix the owner made in
create-verifier/SKILL.md, and it needed the M5 gate to accept it (owner instruction, 2026-08-31).

scripts/sync-harness-skills.py now compares {harness}/skills/{path} as a single self-reference
token and --fix retargets it per destination. Everything else still has to match byte for byte,
and the rule is tightened in two ways so it cannot become a hole:

  • Each copy must point at itself. A .cursor copy naming the .codex tree is drift and the
    gate says so — normalizing all three to one placeholder would have let that pass silently.
  • Only a path that continues past the tree root counts, so prose that lists the three directories
    (harness-sync's own SKILL.md does) is compared literally instead of being mangled into
    ".cursor/skills, .cursor/skills and .codex/skills".

Four cases added to scripts/test-sync-harness-skills.py pin all of it, including CRLF
preservation — reading with universal newlines would otherwise have rewritten every CRLF skill file
to LF as a "sync". While the mechanism was in hand, the stale .cursor/...control.ps1 pointers in
verify-openloop/SKILL.md and features/README.md were retargeted the same way.

docs/DEFINITION_OF_DONE.md M5 and the harness-sync skill document the exception.

Also

  • INVENTORY.md in all three trees: Onboarding LET'S GO! and Record / stop video are now
    automated 2026-08-31, with a note saying what automated <date> means (a loop drives it and was
    seen passing that day) versus mapped.
  • Gate 5b's marker pattern had no underscore in its character class — written when onboarding was
    the only loop, it read the correct PASS loops=onboarding,record_clip as a missing marker. A
    loop's name is its {name}_loop.py stem, i.e. a Python module name, so the class was widened.
    Exit code and the anchored prefix are still both required.
  • Gate 5b still reads "Onboarding loop" though it now runs both loops. Deliberate:
    scripts/hooks/require-sweep.mjs reads the receipt's onboardingLoop key, and renaming the gate
    renames that key. Worth a follow-up that changes both together.

Related Issue

No issue — verifier coverage from features/INVENTORY.md, follow-up to #162.

Type of Change

  • 🧪 Test/Tooling: a new autonomous verifier plus the harness gate that ships it
  • 📝 Documentation: DoD M5, harness-sync, record-clip recipe, INVENTORY

How Has This Been Tested?

  • Manual Verification: record_clip_loop.py run to a green PASS seven times across two
    Pixel_8_API34 AVD states, each time confirmed by reading the evidence rather than the exit code:
    too-short.xml carries the snackbar, mid-recording.xml carries 1s / 30s, mid-trim.xml /
    cap-trim.xml carry TRIM YOUR VIDEO, and the logcat files carry the finalize lines. Scratch
    directory empty afterwards. Final healthy run: too-short=attempt1 mid=3267ms cap=31267ms cap-chip-peak=28s chip-samples=7 took=75s.

  • Automated Tests: python scripts/run-verification-loops.py --changed
    PASS loops=onboarding,record_clip, which also proves the retargeted .cursor copies are
    functional, since that is the tree the runner discovers from.
    python scripts/test-sync-harness-skills.py → 15/15 cases.
    python scripts/sync-harness-skills.py --check → in sync, 42 files x 3 harnesses.

  • Sweep: GREEN on d3caa1f, including gate 5 at 121 instrumented tests / 0 failures and gate
    5b at PASS loops=onboarding,record_clip.

    One note for the record, since it burned most of the wall time here and is worth knowing before
    anyone re-runs this: an earlier sweep's instrumented-test run died with
    INSTRUMENTATION_ABORTED: System has crashed under host memory pressure (committed 65 GB against
    a 70 GB limit). That left the AVD's Android runtime broken in a way adb reboot does not
    clear — every subsequent loop failed with System UI isn't responding (package android) in the
    captured dump, which is the verifier reporting the host honestly rather than the app failing. A
    cold boot (-no-snapshot-load) fixed it, after which the loops pass under the same concurrent
    sweep load. The gotcha is written into record-clip.md.

Checklist

  • 🧪 My changes have been verified locally and work as expected.
  • 🔍 I have performed a self-review of my own code.
  • ✍️ I have commented my code, particularly in hard-to-understand areas.
  • 📖 My changes generate no compile warnings or errors (allWarningsAsErrors is on).
  • 🧹 Pre-PR sweep GREEN on the final commit (.\scripts\pre-pr-sweep.ps1
    build/sweep-receipt.json). Inspect Code export: SKIPPED because: run with
    -SkipInspectCode — Engine 2 needs the IDE run with the "OpenLoop Tracked" scope, which is not
    available headless on this machine.
  • 🏪 Play-facing docs aligned: no permission, data-collection, storage or user-facing change
    — verifier and repo tooling only.
  • 🧹 The git branch is clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SKsnmxgAAPJBMztyREhkoQ


Note

Low Risk
Changes are confined to verification scripts, skill docs, and harness sync rules—no shipped Android app logic in the diff.

Overview
Adds an emulator-driven record_clip_loop.py verifier for Record / stop video, wired into the feature recipe and marked automated 2026-08-31 in INVENTORY.md across all three harness trees. It exercises three shutter outcomes (sub-400 ms too-short with logcat-timed dump, mid recording → Trim + scratch clip, and 30 s cap without stop), asserts the countdown chip stays {n}s / 30s (not mm:ss, issue #154), and discards clips so the gallery stays clean.

Refactors verifier plumbing: new shared verify_common.py (adb/uiautomator, package-filtered nodes, safer dump_ui, parameterized evidence_dir) and a slimmer onboarding_loop.py that imports it—same pattern in .claude, .cursor, and .codex copies.

Harness M5 alignment: harness-sync and create-verifier now treat each tree’s self-referencing skill paths as the only allowed drift (each copy points at its own .{harness}/skills/...); verify-openloop docs/commands retarget control.ps1 and sibling skills per harness instead of hard-coding .cursor or .claude everywhere.

Reviewed by Cursor Bugbot for commit 0b2fca6. Bugbot is set up for automated code reviews on this repo. Configure here.

stozo04 and others added 3 commits August 31, 2026 21:11
Adds `record_clip_loop.py`, the installed-APK verifier for `features/record-clip.md`.
It drives all three outcomes of a shutter tap on an emulator and asserts the countdown
chip's format on every sample:

- too-short: both taps in one `adb shell` land inside the 400 ms minimum, so the clip is
  discarded, "That was quick!" shows and no editor opens.
- mid-length: Trim opens, a scratch clip is on disk, logcat carries `Capture finalized (Nms)`.
- cap: no stop tap at all — the 30 s cap finalizes the recording by itself.

Every countdown sample must read `<seconds>s / 30s` and never a `mm:ss` clock, which is the
issue #154 regression a Compose test that only checks the chip exists cannot see. Both clips
are thrown away through the product's own Discard dialog, so nothing is saved.

Two findings the loop is shaped around, both from measuring on a Pixel_8_API34 AVD:

- The snackbar lasts 4 s and a uiautomator dump takes ~5 s, so dump-polling always loses that
  race. The loop waits on logcat and then dumps once, into the open window.
- The elapsed counter accumulates 33 ms per tick rather than reading a clock, so an emulator
  that cannot keep cadence reaches the cap in ~2.5 min of wall time and produces a 143 s file.
  The cap is therefore asserted through the chip and the finalize, never a wall-clock ceiling.

The plumbing both loops shared is extracted to `verify_common.py` (the second real loop is the
point create-verifier names for extracting), and `onboarding_loop.py` now imports it — re-run and
still green. `dump_ui` no longer reads a stale XML when uiautomator fails, and nodes carry their
package so the system status bar's clock cannot trip the mm:ss assertion.

Harness trees: a skill that points at its own tree has to name a different directory in each copy,
or two of the three send their LLM to a path it cannot read (owner instruction, 2026-08-31). The
M5 gate now compares `<harness>/skills/<path>` as one self-reference token and `--fix` retargets it
per destination; everything else still has to match byte for byte, and each copy must point at
ITSELF. Prose that lists the three directories is left literal. Four cases added to
`test-sync-harness-skills.py` pin all of it, CRLF preservation included.

Verified: `run-verification-loops.py` → `PASS loops=onboarding,record_clip` on emulator-5556.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SKsnmxgAAPJBMztyREhkoQ
A loop's name is its `<name>_loop.py` stem, and those are Python module names — `record_clip`,
not `record-clip`. Gate 5b's marker pattern was written when `onboarding` was the only loop and
its character class has no underscore, so the runner printing the correct
`PASS loops=onboarding,record_clip` with exit 0 was read as a missing marker and failed the gate.

Widening the class is fixing the gate's parser to accept the marker the runner documents and
emits, not relaxing what the gate proves: exit code and the anchored `PASS loops=` prefix are
both still required.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SKsnmxgAAPJBMztyREhkoQ
…ip sample

The cap scenario required the countdown to have reached 20s in the final dump before Trim opened.
That number measures the polling interval, not the product: a dump costs seconds, so the faster
the device runs the counter the further from the cap the last sample sits. It held at 28-29s on a
loaded AVD and failed at 17s on the same AVD cold-booted, which is the wrong way round for a gate.

The cap is now asserted by what does not move with sampling: no stop tap is sent, no error
finalize appears, Trim opens anyway, and the finalized clip is at least 25 s. A recording ended by
anything other than the cap fails all four. The chip is still asserted on every sample for its
format, which is what issue #154 broke; the scenario additionally requires that the chip rendered
at all during a full-length recording, and samples every 3 s instead of 5 so the format is checked
over more of the recording.

Measured on a healthy cold-booted Pixel_8_API34 AVD: the cap produces a 31.2 s clip and the whole
loop runs in 75 s. The same AVD under host memory pressure took 219 s and produced a 143 s clip
for the same cap — the counter accumulates 33 ms per tick rather than reading a clock, so a device
that cannot hold the cadence overruns. Both are the cap firing, which is why no wall-clock ceiling
is asserted. Docs updated to lead with the healthy numbers, plus a gotcha for the failure mode
that cost the most time here: under memory pressure the AVD's own system ANRs, every recipe fails
at step one, and `adb reboot` restores the same broken runtime — it needs a cold boot.

Verified: `run-verification-loops.py` -> `PASS loops=onboarding,record_clip` on emulator-5584.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SKsnmxgAAPJBMztyREhkoQ

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Stale logcat misclassifies too-short retries
    • Added check to distinguish stale FINALIZED_RE matches from genuine ones by verifying absence of TOO_SHORT_LOG_RE pattern.

You can send follow-ups to the cloud agent here.

Reviewed by Cursor Bugbot for commit d3caa1f. Configure here.

Comment thread .cursor/skills/verify-openloop/helpers/record_clip_loop.py
When scenario_too_short retries after a full clip, clear_logcat may not fully clear stale 'Capture finalized' lines. This caused genuine too-short attempts to be misclassified as full clips, triggering discard_clip on the camera/snackbar screen instead of the editor, causing failures.

Fix: Check for both FINALIZED_RE and absence of TOO_SHORT_LOG_RE. If both patterns are present, the finalize is stale and this attempt was actually too-short.
@stozo04
stozo04 merged commit 316d5a0 into main Sep 1, 2026
3 checks passed
@stozo04
stozo04 deleted the feature/record-clip-verifier branch September 1, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants