Skip to content

Build deterministic physical HID fixture and capture lab - #1238

Draft
malpern wants to merge 102 commits into
masterfrom
codex/pico-hid-fixture
Draft

Build deterministic physical HID fixture and capture lab#1238
malpern wants to merge 102 commits into
masterfrom
codex/pico-hid-fixture

Conversation

@malpern

@malpern malpern commented Jul 28, 2026

Copy link
Copy Markdown
Owner

What changed

  • adds the ESP32-S3/Pico physical USB HID fixture, secure Wi-Fi control plane, dual-slot OTA, display health, animated device UI, safety releases, and QEMU coverage
  • adds the native KeyPath HID Capture Jig with a fail-closed focus oracle, exact event ledger, resource admission, responsive UI, and persisted JSON evidence
  • adds combined physical runners plus a Shift lead/hold/release diagnostic matrix correlated with bounded firmware traces
  • expands the installer/runtime lab, scenario campaign tooling, managed clone support, and lab documentation
  • routes CLI start/stop/restart service operations through the privileged helper instead of invoking system launchd control from an unprivileged process

Why

KeyPath needs repeatable end-to-end evidence that real installer/runtime builds correctly process physical keyboard input, including high-rate and resource-pressure regressions. Previous TextEdit and scheduler-only checks could not prove focus, exact output, or safe release. The new Jig and fixture preserve independent machine-readable evidence and fail closed when focus or host readiness is invalid.

Validation

  • Scripts/lab/pico-hid-fixture-tool test (host/core/client/runner/matrix plus ESP32-S3 QEMU smoke)
  • native HID Capture Jig suite: 26 tests
  • KeyPath privileged-helper and runtime-health regression suite: 56 focused tests
  • production firmware build and authenticated OTA verification on Waveshare ESP32-S3 build 4d1cb1cb54c4
  • connected-board status: valid OTA slot, USB mounted, Wi-Fi connected, splash complete, display heartbeat healthy
  • admitted three-cell physical Shift smoke matrix: baseline, 4 ms lead, and 8 ms lead all delivered exact 24/24 output with valid focus, released keys/modifiers, complete firmware traces, and zero late reports
  • full 5x5 physical Shift lead/release matrix: 25/25 valid cells, 1,500/1,500 characters captured, 4,000 firmware trace reports, valid focus and clean final releases throughout, and zero late reports
  • full-matrix diagnosis: 10 Shift demotions were confined to four of five zero-lead cells; all 20 cells and 1,200 characters with a 2 ms-or-greater modifier lead were exact

Pending physical acceptance

This remains a draft until the remaining person-dependent physical gates are complete. Real USB HID input cannot target a background app safely. Remaining gates are:

  • final visual confirmation that the repaired splash completes
  • USB disconnect/reconnect and intentional Wi-Fi-loss safe-abort checks

Excluded focus-, memory-, load-, and network-admission attempts are documented as harness admissions, not KeyPath outcomes.

malpern added 30 commits July 24, 2026 20:14
@claude

claude Bot commented Jul 28, 2026

Copy link
Copy Markdown

Automated review of the supplied diff (note: the diff payload was truncated before reaching Sources/KeyPathCLI/** and Sources/KeyPathCLISupport/**, so PrivilegedOperationsRouter.swift, SystemUninstallCommand.swift, CLIReportModels.swift, ServiceBootstrapper.swift, and the related test changes were not visible for this pass — please get those specifically reviewed).

Findings from the visible portion (Scripts/lab/**, Scripts/build-and-sign.sh):

  1. Scripts/build-and-sign.shSKIP_DEPLOY=1 still publishes the help-content website. The new if [ "${SKIP_DEPLOY:-0}" = "1" ]; then ... else <stop/replace/restart KeyPath> fi block closes right before the # Publish help content to website section, so the GHPAGES_DIR/SKIP_WEBSITE publish step runs unconditionally regardless of SKIP_DEPLOY. If the intent of SKIP_DEPLOY is "produce the signed artifact only, don't touch anything else on this machine" (as the added comment implies — "must not replace or restart the operator's installed app"), publishing to the local gh-pages worktree during a matrix build seems like an unintended side effect. Worth confirming this is intentional, or gate the website publish on SKIP_DEPLOY too.

  2. Scripts/lab/damage-kanata-service isn't idempotent under set -euo pipefail. It unconditionally runs sudo -n /bin/launchctl bootout system "$plist" (or the label form) with no fallback if the job is already unloaded. launchctl bootout exits non-zero when there's nothing to boot out, and with set -e that aborts the whole fixture script instead of treating "already stopped" as success. If any scenario ever calls this fixture when Kanata isn't currently loaded, the script dies instead of confirming the degraded state.

  3. Scripts/lab/desktop-bootstraptrap ... EXIT set inside install_desktop_tools. The function does trap "/bin/rm -rf '$temp'" EXIT, which is a shell-global trap, not function-scoped. That's fine today since nothing else in the script sets an EXIT trap, but it's a fragile pattern if a future edit adds another cleanup trap elsewhere in this script — the later one silently wins and the earlier cleanup is lost.

Nothing else in the visible diff (the new HIDCaptureCore/CaptureSession Swift logic, assert-runtime-state, assert-uninstalled-state, hid-capture-jig-client) raised correctness concerns on read-through.

@malpern

malpern commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

Latest reliability update (commit 592c9aa):

  • adds a two-step, compiled-in offline demo (top power, then touch) so Wi-Fi is not on the critical start path
  • caches the signed Jig app and adds hid-capture-jig-tool demo with in-Jig operator instructions
  • preserves an atomic inconclusive artifact and waits through the bounded capture window when control/abort is unconfirmed
  • adds ESP reset/heap/Wi-Fi/HTTP diagnostics, server restart recovery, no power-save, and a higher-priority HTTP task
  • adds a no-HID control soak that separates mDNS discovery from endpoint latency

Validation after merging current master:

  • fixture C core: pass
  • Python client/tool/campaign/runner/Shift suites: 41 tests pass
  • Swift HID Capture Jig: 26 tests pass
  • ESP32-S3 QEMU smoke: pass
  • production ESP-IDF build: pass (59% app partition free)
  • live pre-update control soak: 20/20, p95 206.745 ms, max 219.339 ms, no reset/build discontinuity

Local review gate selected the enforced remote claude-review; waiting for that check before merge. No physical HID was emitted during this update.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review notes (based on the supplied diff, which was truncated after ~60000 bytes — the actual Sources/KeyPathCLI/Core/PrivilegedOperationsRouter.swift, SystemUninstallCommand.swift, ServiceBootstrapper.swift, and the associated CLI test changes were not visible to me. Given this repo's own emphasis on postcondition-verified installer/uninstall mutations, that part of the diff deserves an explicit separate look before merge.)

Findings from the visible portion:

  1. Possible wrong upstream source for downloaded binaries (Scripts/lab/desktop-bootstrap): the new bootstrap fetches Peekaboo and mcporter directly from GitHub releases at github.com/openclaw/Peekaboo and github.com/openclaw/mcporter, whereas the code being replaced installed them from the steipete/tap/peekaboo and steipete/tap/mcporter Homebrew taps. The pinned SHA-256 checksums make tampering-in-transit unlikely, but if openclaw isn't actually the intended/trusted publisher for these tools (vs. a typo, fork, or unrelated org), this silently changes the trust root for two binaries that get code-signed as a background LaunchAgent-launched app and executed on every lab VM. Worth a quick confirmation that this is the intended source before merge.

  2. Behavior change in consent-prompt handling (Scripts/lab/desktop-bootstrap): the old code detected the Peekaboo "Allow" consent sheet and automatically clicked it. The new code only detects the prompt (via an osascript -l JavaScript check) and exits with code 4 plus a capture-consent-checkpoint.json artifact — it no longer clicks anything. That's a reasonable move if Apple's TCC path requires human-driven consent, but it means any orchestrator (keypath-lab, remote.sh, scenario runners) calling desktop-bootstrap must now explicitly special-case exit code 4 as "waiting for a human," rather than treating any non-zero exit as a hard failure. Please confirm the callers were actually updated to handle this new exit code (not visible in the truncated diff) rather than just failing the scenario.

No other correctness/security issues stood out in the visible Python/Swift/shell lab tooling (assert-runtime-state, assert-uninstalled-state, CaptureSession.swift, hid-capture-jig-client, etc.) — logic there looked internally consistent.

@malpern

malpern commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

Post-update hardware control evidence: OTA build f92209f3ea52 verified healthy in valid slot ota_1. A 120-request no-HID soak passed 120/120 with 28.153 ms median, 93.715 ms p95, 105.845 ms max, zero request failures/resets/build changes/server restarts, max firmware handler latency 7.465 ms, and 8.33 MB free heap. Artifact: ~/.local/state/keypath-hid-fixture/control/control-soak-20260729T014347Z.json. Physical button/touch/demo-output validation remains intentionally deferred until an exclusive desktop window.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review scope note: the diff supplied to me was truncated after ~60k bytes (cut off partway through Scripts/lab/hid-capture-jig/...KeycapBurstModel.swift), so I did not see the changes to Sources/KeyPathCLI/.../PrivilegedOperationsRouter.swift, SystemUninstallCommand.swift, Sources/KeyPathCLISupport/CLIReportModels.swift, ServiceBootstrapper.swift, or the associated tests. Those should get a separate look.

Finding — Scripts/build-and-sign.sh: SKIP_DEPLOY=1 doesn't gate the website-publish step

The new SKIP_DEPLOY block wraps the "stop/replace/restart the installed app" section in if [ "${SKIP_DEPLOY:-0}" = "1" ]; then … else … fi, closing the fi right before GHPAGES_DIR=.... The stated intent (per the added comment) is that a matrix build "must not replace or restart the operator's installed app." But the subsequent "Publish help content to website" block is outside that if/else and keeps its original guard ([ -d "$GHPAGES_DIR" ] && [ "${SKIP_WEBSITE:-0}" != "1" ] && [ "${SKIP_NOTARIZE:-}" != "1" ]), so it still runs regardless of SKIP_DEPLOY.

Since matrix/lab builds want "the exact signed artifact" they presumably don't set SKIP_NOTARIZE=1, so on any machine where .worktrees/gh-pages happens to exist (e.g. a release engineer's box also used to produce matrix-test artifacts), running SKIP_DEPLOY=1 ./Scripts/build-and-sign.sh would still publish/copy help content toward the live website — an unintended side effect for a flag meant to make the build a no-op with respect to the operator's environment. Consider also gating the website-publish block on SKIP_DEPLOY, or explicitly documenting that it's intentionally excluded.

No other actionable correctness/security issues found in the reviewed portion (the new assert-runtime-state/assert-uninstalled-state/desktop-bootstrap lab scripts look reasonable; checksums in desktop-bootstrap are valid 64-hex-char SHA-256 values).

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (note: it was truncated at 60000 bytes, cutting off before Sources/KeyPathCLI/.../PrivilegedOperationsRouter.swift, SystemUninstallCommand.swift, ServiceBootstrapper.swift, CLIReportModels.swift, Scripts/lab/keypath-lab, Scripts/lab/remote.sh, and most of HIDCaptureJig/main.swift — those are worth a separate pass, especially given AGENTS.md's postcondition-verification requirements for install/uninstall/service-lifecycle code).

One plausible flake in the portion I could review:

Scripts/lab/hid-capture-jig-tool demo subcommand — focus/arm race

"$client" reset >/dev/null
"$client" focus >/dev/null
"$client" arm --run-id offline-demo --expected "$expected" ...

focus and arm are sent as two independent synchronous file-RPC calls with nothing waited on between them. CaptureSession.arm() (Scripts/lab/hid-capture-jig/Sources/HIDCaptureCore/CaptureSession.swift) hard-fails immediately if focused is false at the moment arm is processed:

guard !runID.isEmpty, runID.count <= 80, !expected.isEmpty,
      timeoutMs >= 250, timeoutMs <= 300_000,
      settleMs >= 50, settleMs <= 5000,
      focused
else { ... state = .failed; return false }

If the app's window/app-activation focus notification lands even one runloop tick after the focus RPC returns "ok", the immediately-following arm call will read stale focused=false and the whole demo run fails non-deterministically — independent of anything the operator does. Consider having arm poll/wait briefly for focus (or having the client poll status for focused: true before sending arm) instead of assuming synchronous ordering.

Everything else reviewed (build-and-sign.sh's new SKIP_DEPLOY guard, the new assert-* fixtures, desktop-bootstrap's checksum-verified tool installation) looks sound.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (note: it was truncated after ~60KB, so remote.sh, physical-hid-capture-run, scenario-matrix-runner, and most of the Swift HID-capture-jig sources beyond KeycapBurstModel.swift weren't visible to me — worth a follow-up pass on those).

One actionable finding in the portion reviewed:

Scripts/lab/desktop-bootstrap — version/checksum pins are bypassed once a binary already exists. install_desktop_tools() gates each download on [[ ! -x "$local_bin/peekaboo" ]] (same pattern for mcporter, uv), not on the binary's actual version/hash. If a lab host is reused across script revisions (rather than being a freshly provisioned VM every time), a stale binary from a prior bootstrap run will silently stick around even after PEEKABOO_VERSION/PEEKABOO_SHA256 etc. are bumped in this file — the checksum verification in download_verified() never runs against it. Consider checking the installed binary's hash (or a version marker file) against the pinned constant and re-downloading on mismatch, so bumping the pinned version actually takes effect on existing hosts.

Everything else I could see — the SKIP_DEPLOY wrapping in build-and-sign.sh, the new assert-runtime-state/assert-uninstalled-state verifiers, and the hid-capture-jig-client/-tool command protocol — looked structurally sound; no other correctness/security issues stood out.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Correctness bug: CaptureSession.arm() failure isn't sticky, so a bad arm can later self-report as "passed"

Scripts/lab/hid-capture-jig/Sources/HIDCaptureCore/CaptureSession.swift

In arm(), the validation-guard failure path only sets state = .failed — it never sets terminalFailure = true, and deadlineAtNs/settleNs are left at their defaults (deadlineAtNs = nil, settleNs = 250_000_000) because the success-path assignments are skipped:

guard !runID.isEmpty, runID.count <= 80, !expected.isEmpty,
      timeoutMs >= 250, timeoutMs <= 300_000,
      settleMs >= 50, settleMs <= 5000,
      focused
else {
    self.runID = runID
    self.expected = expected
    self.focused = focused
    state = .failed
    return false
}

If runID is non-empty in this failure case (e.g. arm rejected only because focused == false, or timeoutMs/settleMs out of range), subsequent record() calls pass the !runID.isEmpty guard and reach evaluate(). Since terminalFailure is still false, the early-out if terminalFailure { state = .failed; return } never fires, so evaluate() falls through to its normal pass/fail logic. Because deadlineAtNs is nil, the timeout branch never triggers, but the settle branch:

if received == expected, pressedKeyCodes.isEmpty, activeModifiers == 0,
   let lastEventAtNs, nowNs >= lastEventAtNs,
   nowNs - lastEventAtNs >= settleNs
{
    state = .passed
}

can still trigger once received catches up to expected and ~250ms (the default settleNs) elapses with no keys held — silently flipping a rejected/invalid arm from .failed to .passed. This would let a capture run that was armed with invalid parameters report a false PASS to callers polling status/wait in hid-capture-jig-client/hid-capture-jig-tool, undermining the acceptance signal this fixture exists to provide.

Suggested fix: set terminalFailure = true in the guard-else branch (or gate record()/evaluate() on a distinct invalidArm flag) so a failed arm() can never transition to .passed.


Note: this diff is very large (Swift capture-jig core, Python/zsh lab scripts, ESP32/Pico firmware in C, etc.) and was truncated in the review context after ~60KB. I reviewed the visible portion in depth (Swift CaptureSession, the Python assert-runtime-state/assert-uninstalled-state/hid-capture-jig-client scripts, and the desktop-bootstrap/build-and-sign.sh shell changes) and found no other high-confidence issues there. The remainder of the diff (firmware sources, additional test files, remote.sh, scenario-matrix-runner, etc.) was not visible to this review and should get a separate pass.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (note: it was truncated after Scripts/lab/hid-capture-jig/Sources/HIDCaptureCore/CaptureSession.swift, so I could not see the actual content of the product-code changes later in the file list — PrivilegedOperationsRouter.swift, SystemUninstallCommand.swift, ServiceBootstrapper.swift, CLIReportModels.swift, and the CLI test updates. Those are the highest-risk changes in this PR since everything else is disposable lab tooling; please make sure they get separate scrutiny, e.g. via ./Scripts/review-gate.sh.).

Within what I could see:

  • Scripts/build-and-sign.sh: the new SKIP_DEPLOY=1 branch wraps the "stop process / replace bundle / restart" block in an if/else/fi, with the closing fi inserted right before the "Publish help content to website" section. The bracing looks correct, but worth double-checking that nothing set inside that now-conditional block (e.g. a deploy path/version variable) is read later in the script when SKIP_DEPLOY=1 skips it — that portion of the file wasn't in the diff, so I couldn't confirm.

  • Scripts/lab/desktop-bootstrap: the old flow auto-clicked "Allow" on the privacy consent sheet; the new flow instead detects the consent dialog via JXA and exits with code 4 (a "checkpoint") for external handling. That's a real behavior change for any caller that expected desktop-bootstrap --install-tools to complete unattended — worth confirming keypath-lab/CI callers were updated to handle exit code 4 (not visible in the supplied diff).

Everything else reviewed (checksum-pinned downloads in desktop-bootstrap, assert-runtime-state, assert-uninstalled-state, CaptureSession.swift state machine) looked internally consistent with no obvious correctness bugs.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (truncated at 60000 bytes out of a 144-file/~18.6k-line diff, so most of the new pico-hid-fixture C/ESP32 firmware and the HIDCaptureJig Swift package beyond CaptureSession.swift were not visible to this review).

Scripts/lab/damage-kanata-service — hard failure if the service is already unloaded

set -euo pipefail
...
if [[ -f "$plist" ]]; then
  /usr/bin/sudo -n /bin/launchctl bootout system "$plist"
else
  /usr/bin/sudo -n /bin/launchctl bootout system/com.keypath.kanata
fi

launchctl bootout exits non-zero when the job isn't currently loaded/registered (e.g. it was already booted out by a prior run, or install-runtime hasn't started it yet). Because the script runs under set -e, that failure kills the script immediately — it never reaches the polling loop below that would otherwise treat "no longer registered" as success (print "damage_fixture\tkanata-service-booted-out"). Net effect: this "damage" fixture can spuriously fail scenario runs purely due to fixture-script fragility rather than an actual runtime state problem. Consider tolerating a non-zero bootout exit (e.g. ... || true) and always falling through to the existing poll/verify loop to determine success/failure from actual launchd state.

No other actionable correctness/security issues found in the portion of the diff reviewed (build-and-sign.sh's new SKIP_DEPLOY if/else appears correctly scoped around just the stop/replace/restart block, and the checksum-pinned desktop-bootstrap downloads use full 64-char SHA-256 values as expected).

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (truncated after 60000 bytes — Swift/C sources for the HID capture jig, pico-hid-fixture firmware, scenario-matrix runner, and keypath-lab changes beyond that point were not covered).

1. Peekaboo/mcporter download source changed orgs (Scripts/lab/desktop-bootstrap)
install_desktop_tools() now pins and downloads from github.com/openclaw/Peekaboo and github.com/openclaw/mcporter, replacing the previous brew install steipete/tap/peekaboo / steipete/tap/mcporter install path referenced elsewhere in the repo (AGENTS.md still documents steipete/tap/peekaboo). SHA256 pinning mitigates tampering-in-transit, but this is a trust-boundary change (individual steipete identity → openclaw org) that isn't called out in the diff. Worth confirming openclaw is genuinely the canonical upstream for these tools before this lands, since a wrong/typo'd org would silently pin and execute an unrelated binary in the lab bootstrap path.

2. Scripts/build-and-sign.sh — verify the new SKIP_DEPLOY conditional scoping
The new if [ "${SKIP_DEPLOY:-0}" = "1" ]; then ... else <stop/replace/restart> fi wraps the local-deploy block, and the added closing fi lands right after the # Publish help content to website comment header but before the GHPAGES_DIR=... check. That means the website-publish logic sits outside the SKIP_DEPLOY guard (runs regardless), while the comment header describing it is now inside the guarded block. Given two fis appear back-to-back with a comment sandwiched between them, please double check both the SKIP_DEPLOY=1 and default paths execute exactly as intended (e.g. bash -n plus a dry run of both branches) — it's easy for a stray/misplaced fi in a script this size to silently change what block a later section belongs to.

3. Scripts/lab/desktop-bootstrap — consent-prompt auto-click removed
The previous implementation parsed Peekaboo's permission JSON and automatically clicked "Allow" on the notification-center consent sheet. The new implementation only detects a screen-capture or private-window-picker consent prompt via JXA and exits with code 4 plus a checkpoint JSON, without clicking anything. If this is an intentional shift to a manual/external-checkpoint model, please confirm the caller(s) (keypath-lab, scenario runners) actually handle exit code 4 by resuming after out-of-band approval — otherwise previously-unattended lab bootstrap runs could now stall or fail where they used to auto-proceed.

No other correctness/security issues found in the reviewed portion (the new assert-runtime-state, assert-uninstalled-state, hid-capture-jig-client, and hid-capture-jig-tool scripts look internally consistent).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant