feat: read a QR code on the page and hand the human the link

CHANGELOG.md

@@ -7,8 +7,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased] — 0.6.0
 
+QR passthrough, and typed errors on everything `raiseHand` throws.
+
+A device-change check draws a QR code and asks for a phone — and handraise's
+human is holding that phone, looking at the code on its screen. A phone cannot
+scan itself, so until now this needed a second device. The agent reads the code
+off the page instead and hands the human the link.
+
+The QR half is additive: no existing call, type or outcome changes. The errors
+half is not entirely — a page that is already dead now throws instead of
+returning `disconnected`. Both are under **Changed**.
+
 ### Added
 
+- **A `Scan QR` key on the phone, in takeover mode.** It asks the agent to read
+  the QR codes on the page; the answer arrives as a sheet showing what each one
+  said, with **Open in new tab** and **Copy**. The button is disabled while a
+  scan is in flight and releases itself if no answer comes.
+- **Two protocol messages**: human→agent `{ type: "scanqr" }`, and agent→human
+  `{ type: "links", links: ScannedLink[], source: "qr" }` — always sent, with
+  an empty list when nothing decoded, because silence reads as a broken button.
+  The relay routes `scanqr` in takeover mode only.
+- **`scanQrLinks(png)`, `createQrScanner()` and `OPENABLE_SCHEMES` are
+  exported.** The decoder is usable without a human: hand it a PNG screenshot,
+  get back up to two `{ text, kind }`. `scanQrLinks` is synchronous;
+  `createQrScanner()` is the same decode on a worker thread, which is what a
+  handoff uses — measured, a 4K screenshot held the event loop for 2132 ms and
+  now holds it for 1 ms.
+- **`kind` is `"url"` only for `http:`, `https:` and `mailto:`**, with no
+  control or bidi characters and no credentials in the authority. `tel:` and
+  `otpauth:` were openable in earlier drafts of this release and are not: a
+  dialler control sequence and an authenticator enrolment are actions rather
+  than pages, and neither is worth one tap from a page nobody vetted. They are
+  still decoded, shown in full and copyable, under a label that names them.
+- **The phone applies that whole rule again** rather than trusting a label that
+  crossed a socket a stranger holding the link can write to, and an openable
+  link is displayed, anchored and copied as the address it resolves to, with
+  the host as the loud part — so a homograph host cannot show one address and
+  open another.
+- **The PNG decoder refuses what it cannot have produced.** Dimensions,
+  compressed size and the exact inflated length are all bounded by the header
+  before a byte is decompressed, chunk boundaries are walked, and IEND is
+  required. CRCs are deliberately not checked, and the ADR says why.
+- **The relay bounds its own ingress**: 4 KiB per human message enforced before
+  the parse, the scan floor enforced there as well as in the core, and the
+  human socket held while the agent's is backpressured — with terminal answers
+  delivered first, never dropped.
+- **`HandoffEvent.qrScans` and `HandoffEvent.qrHits`.** Two new **required**
+  number fields on the wide event — additive for callers, who receive the
+  event rather than construct it, but a TypeScript consumer that builds a
+  `HandoffEvent` literal in a test will need them. Both are 0 in approval mode,
+  which offers no scan. `qrScans - qrHits` is the number worth watching.
+- **`jsqr` as a runtime dependency** (pure JavaScript, no dependencies of its
+  own), and a PNG decoder written against `node:zlib` in `src/core/png.ts`.
+  `BarcodeDetector` does not exist in Solari's Chromium, so the decode happens
+  in the agent process — never in the remote page, whose JavaScript belongs to
+  whoever the agent got stuck on.
+- **[ADR 0008](docs/adr/0008-qr-passthrough.md)** and
+  **[measurement 05](docs/measurements/05-qr.md)**, reproducible with
+  `bun --env-file=.env scripts/measure-qr-decode.ts`.
 - **Typed errors: `HandraiseError`, `HandraiseErrorCode`, `isHandraiseError`.**
   Everything `raiseHand` throws now carries a `code` you can branch on —
   `missing_api_key`, `invalid_mode`, `empty_action`, `browser_unusable`,
@@ -59,6 +116,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `node20`). It always was the floor: `@solarisdk/browser` and the patchright
   runtime it wraps require Node 20, so a Node 18 install never worked; the
   package just did not say so.
+- **`src/relay/guest/server.js` names its wire vocabulary.** `MSG` and `MODE`
+  replace the bare strings the untyped relay compared against, and the mobile
+  page it serves is handed the same object at serve time instead of keeping its
+  own copy. `relay.test.ts` asserts `MSG` against the TypeScript protocol's own
+  unions, so neither side can grow a message alone. No behaviour change.
 - **A page that is already dead is now refused instead of handed off.**
   `raiseHand` used to create a relay, fail on the first CDP call and *return*
   `{ outcome: "disconnected" }`. It now throws a `HandraiseError`
@@ -83,6 +145,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   Apart from the page check above, nothing throws that did not throw before,
   and no outcome became an exception.
 
+### Known limits
+
+- A scan takes a fresh full-resolution `page.screenshot()`, 293 ms p50 measured
+  from Germany. It is rate-limited to one per 2 s in the core.
+- A symbol drawn below about 120 CSS pixels does not decode. The live cast
+  frame — 800 px, JPEG quality 60 — fails well before that, which is why the
+  scan does not reuse it.
+- A code the page drew at a resampled size can be sharp and still not be found
+  on the first pass, so a scan looks again at 2x and then at four overlapping
+  corners. A page with no code at all pays all three, about 320 ms of CPU.
+- Two codes on one screen need a tiled second pass to be found at all; three or
+  more are not attempted.
+- **reCAPTCHA itself is untested.** Its demo never served the scan-to-verify
+  variant, which Google shows at its own discretion. The mechanism is proven
+  end to end in the live e2e against a page that behaves the same way.
+
 ## [0.5.1] - 2026-09-02
 
 Republish of 0.5.0 with no code change. 0.5.0 was published to npm and

README.md

@@ -206,15 +206,53 @@ zoom and pan yourself; double-tap toggles between zoomed and fit. Typing goes
 straight into the focused field, character by character — and if that field is
 a one-time code, the phone offers the SMS code it just received.
 
-Four keys under the input, because a phone's virtual keyboard cannot be trusted
-to send them:
+Under the input, four keys a phone's virtual keyboard cannot be trusted to
+send, and one that asks the agent a question about the page:
 
 | Key | What it does |
 |---|---|
 | ⌫ | Delete one character in the remote field |
 | ⇥ | Move to the next field |
 | ⏎ | Submit / press Enter |
 | Clear | Empty the focused field (select-all + backspace; disabled while nothing is focused, and kept well away from ⌫) |
+| Scan QR | Read the QR codes on the page and show what they say |
+
+### QR codes on the page
+
+Some walls ask for a *second device*: reCAPTCHA's "scan to verify", a WhatsApp
+Web login, an authenticator enrolment. The human is holding the phone the site
+wants — and the code is on that phone's screen, so it cannot be scanned.
+
+**Scan QR** asks the agent instead. It takes a fresh full-resolution screenshot
+of the page, decodes it, and sends back what each code said. The phone shows the
+link in full and offers **Open in new tab** — so the link is opened on the
+phone, which is the device the site was asking for. Takeover mode only, one scan
+per 2 seconds, and a symbol below about 120 CSS pixels will not decode — scroll
+or zoom the remote page and scan again.
+
+The code came off a page nobody vetted, so:
+
+- Only `http`, `https` and `mailto:` get an **Open** button. Everything else —
+  `javascript:`, `data:`, `blob:`, `content:`, and anything carrying an
+  invisible character — is shown as text with a Copy button and no link. The
+  agent classifies it and the phone applies the same rule again, because the
+  handoff URL is a bearer credential and the socket behind it takes messages
+  from anyone holding it.
+- **`tel:` and `otpauth:` are shown and copyable, never opened.** A `tel:` code
+  can carry a dialler control sequence, and an `otpauth:` code enrols a TOTP
+  secret in your authenticator. Both are one tap and hard to take back, so the
+  sheet names them ("Phone number", "Authenticator secret") and you hand them
+  to the right app yourself.
+- An openable link is shown **as the address it opens**, with the host as the
+  loud part of it. `https://аpple.com` with a Cyrillic а reads as apple.com and
+  goes to `xn--pple-43d.com`; the sheet shows the second one and says the code
+  wrote it differently.
+- The agent process never fetches any of it, and never decodes on its own event
+  loop: the decode runs on a worker thread, so a handoff stays answerable while
+  it happens.
+
+Measured: [`docs/measurements/05-qr.md`](docs/measurements/05-qr.md); the
+decisions are in [ADR 0008](docs/adr/0008-qr-passthrough.md).
 
 Below that, two ways out. **✋ Hand back** ends the handoff as `resolved` — one
 tap, the agent continues. **I can't do this** ends it as `aborted` — the agent
@@ -289,6 +327,21 @@ await raiseHand(page, {
 | `timeout` | both | Nobody answered within `timeoutMs`. |
 | `disconnected` | both | The browser session died mid-handoff. |
 
+### `scanQrLinks(png): ScannedLink[]`
+
+The decoder behind the phone's **Scan QR** button, exported so an agent can
+read a code without asking a human. Takes the bytes of a PNG screenshot,
+returns up to two `{ text, kind }` — `kind: "url"` only for a scheme in
+`OPENABLE_SCHEMES`, which is also exported. It reads and classifies; it never
+opens anything.
+
+```ts
+import { scanQrLinks } from "handraise"
+
+const codes = scanQrLinks(await page.screenshot({ type: "png" }))
+if (codes[0]?.kind === "url") console.log(codes[0].text)
+```
+
 ### Errors
 
 `raiseHand` throws only before the handoff URL exists — while nobody has been
@@ -369,8 +422,10 @@ handraise brings the same handoff to Solari browsers, which have no native live
 view (Solari's VNC is desktop-only), as a portable library instead — less
 polished, and it works where those don't. What the hosted live views do not
 have is the second mode: an approval is a yes-or-no on one screenshot, no
-live session exposed at all, answerable from a chat channel. Its scope stops
-at the handoff, not wall detection
+live session exposed at all, answerable from a chat channel. Nor do they have
+an answer to a device-change check — a QR code a phone is asked to scan, on
+the phone's own screen — which handraise reads off the page and hands over as
+a link. Its scope stops at the handoff, not wall detection
 ([`docs/adr/0005`](docs/adr/0005-handoff-not-wall-detection.md)).
 
 ## Security
@@ -444,14 +499,17 @@ a 2FA, and each handoff consumes one sandbox, destroyed when it ends.
 
 ## Verified how
 
-Benchmark method and raw data: [`benchmarks/`](benchmarks/README.md). The four
+Benchmark method and raw data: [`benchmarks/`](benchmarks/README.md). The five
 platform measurements the design rests on — transport, screencast, input
-injection, session lifetime — are in
+injection, session lifetime, QR decoding — are in
 [`docs/measurements/`](docs/measurements/README.md). The e2e test drives the
 whole loop with no mocks: a Solari browser signs into a TOTP-protected demo app
 ([`test-app/`](test-app/), deployed into a sandbox), hits the 2FA wall, raises
 its hand, a scripted "human" types the code through the real handoff UI, and
-the test asserts the signed-in page — ~6s end to end. Injected events arrive
+the test asserts the signed-in page — ~6s end to end. The same run then drives
+the QR passthrough: the app shows a device-change code, the human asks for a
+scan, and the link that comes back is fetched from outside the browser to reach
+the confirmation page. Injected events arrive
 with `isTrusted: true`.
 
 ## Limitations (v1)
@@ -464,11 +522,12 @@ with `isTrusted: true`.
 - An approval shows the page as it was when the agent asked. If the page
   changes underneath (a session expiring, a redirect), the human is deciding on
   a stale picture — the frame is not refreshed.
-- A verification that shows a QR code to scan (reCAPTCHA's "scan to verify
-  you're human") needs a second screen today: open the handoff link on a
-  laptop and scan it with the phone — the phone cannot scan its own display.
-  Decoding the QR from the live frame and handing the phone the link is
-  planned.
+- The QR passthrough is **untested against reCAPTCHA itself**: its demo never
+  served the scan-to-verify variant, which Google shows at its own discretion.
+  The mechanism is proven end to end against a page that behaves the same way
+  ([measurement 05 §7](docs/measurements/05-qr.md)). A code drawn below ~120
+  CSS pixels does not decode, and three or more codes on one screen are not
+  attempted.
 - TypeScript/Node only for now.
 
 ## Contributing