diff --git a/docs/README.md b/docs/README.md index cd6f59fdf9..369fc2af1b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -35,6 +35,7 @@ This page is the authority map for Maka documentation. Code and contract tests r - [Renderer architecture](../apps/desktop/src/renderer/README.md) - [Evaluation kernel](../packages/eval/README.md) - [Runtime package](../packages/runtime/README.md) +- [Computer Use package](../packages/computer-use/README.md) - [UI package architecture](../packages/ui/README.md) - [Security policy](../SECURITY.md) diff --git a/packages/computer-use/README.md b/packages/computer-use/README.md new file mode 100644 index 0000000000..9d0cb4f736 --- /dev/null +++ b/packages/computer-use/README.md @@ -0,0 +1,122 @@ + + +# `@maka/computer-use` + +`@maka/computer-use` adapts Maka's Runtime-owned Computer Use contracts to the +native `maka-cu` executor. It owns backend selection, the executor process and +protocol lifecycle, host-side result decoding, display snapshot mapping, and +the cursor-overlay hook. Runtime owns the model-facing tools and session state; +Desktop supplies the executable and presentation dependencies. + +## Public seam + +The package exposes one root entry point through `src/index.ts`: + +- `selectComputerUseBackend()` selects the available executor and builds the + Runtime tool set. `CU_BACKEND_IDS` currently contains only `maka-cu`. +- `createMakaCuBackend()` adapts the native executor to Runtime's + `CuDispatchBackend` contract. +- `MakaCuService` supervises the executor process and owns the JSON-RPC request, + cancellation, restart, and generation lifecycle. +- The `maka-cu-protocol` exports decode and validate `maka.cu/2` envelopes, + snapshots, dispatch results, domain errors, and key chords. +- `resolveCuaDisplaySnapshots()` maps executor screenshots to Electron display + coordinates without guessing when the display geometry is ambiguous. +- `createComputerUseOverlayHook()` projects action lifecycle events to a + presentation-only cursor sink. It does not choose or authorize targets. + +Other packages should import these exports from `@maka/computer-use`, not from +undeclared internal source paths. + +## Current platform boundary + +The shipped selector enables Computer Use only when all of these conditions +hold: + +1. the host platform is macOS (`process.platform === 'darwin'`); +2. the composition supplies a `maka-cu` executable path; and +3. the composition supplies the executable's expected SHA-256 digest. + +On another platform, with missing inputs, or when backend construction fails, +selection fails closed to `backendId: 'none'` with an empty tool set. This +package does not discover, download, or choose an unpinned executable. + +The executable's build, provenance, signing, and distribution status are +separate release concerns. See +[`computer-use-provenance.md`](../../docs/computer-use-provenance.md) rather +than assuming that installing this workspace supplies a runnable binary. + +Cross-platform work is tracked separately: + +- [#3896](https://github.com/apache/maka/issues/3896) — platform abstraction; +- [#3891](https://github.com/apache/maka/issues/3891) — Linux backend; +- [#3785](https://github.com/apache/maka/issues/3785) — Windows executor + hardening and production evidence. + +## Protocol and lifecycle + +The host and executor communicate over line-delimited JSON-RPC using the +versioned `maka.cu/2` protocol. `MakaCuService` verifies that the executable is +usable and checks any configured digest before spawning it, completes a +`host.hello` handshake, and exposes the executor version, capabilities, limits, +and process generation. The product selector always supplies the required +digest. + +Lifecycle and protocol failures remain distinct: + +- `MakaCuLifecycleError` reports unavailable, mismatched, aborted, or + outcome-unknown process states; +- `MakaCuRpcError` reports a JSON-RPC error response for one method; and +- `MakaCuProtocolViolation` reports malformed or contradictory wire data. + +An executor exit releases affected sessions and invalidates their observations. +Requests that may have reached the executor surface as outcome-unknown rather +than being replayed automatically. Runtime must re-observe before another +action. + +## Ownership rules + +- Keep provider-neutral Computer Use types and model-facing contracts in + `@maka/core` and `@maka/runtime`. +- Keep native executor transport, decoding, and lifecycle handling in this + package. +- Keep Electron windows, screen-lock integration, binary provisioning, and + product status UI in `apps/desktop`. +- Add a second backend only after it has a real adapter and platform evidence; + do not widen `CU_BACKEND_IDS` with a placeholder. +- Preserve fail-closed selection and snapshot-bound dispatch. Missing or stale + authority must not fall back to global pointer or foreground input. + +The cross-layer safety and evidence rules live in the +[`Computer Use foundation contract`](../../docs/computer-use-foundation-contract.md) +and [`host events contract`](../../docs/computer-use-host-events-contract.md). + +## Verification + +Install dependencies once at the repository root, then run: + +```sh +npm --workspace @maka/computer-use test +npm --workspace @maka/computer-use run typecheck +``` + +The package tests cover protocol decoding, process lifecycle, backend behavior, +host-event propagation, display mapping, overlay projection, and the cumulative +Computer Use path.