machine-machine · mariuszkreft · Aug 24, 2026 · Aug 24, 2026 · Aug 24, 2026 · Aug 28, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,28 @@
 # Changelog
 
+## [Unreleased] — 2026-08-24
+### Added
+- Section 12b: m2o Desktop Provisioning (RDP + cross-host Guacamole + console/ttyd) — captures the m2/m2.2 topology, path/DNS gotchas, first-boot `unhealthy` wait, expected WARNs on m2.2, the socat-relay pattern for wiring m2.2 desktops into m2's Guacamole, and the console-auth magic-link flow (single real console-auth on m2, m2.2 bridges via socat).
+- `scripts/launch-m2o-desktop.sh` — idempotent one-command wrapper around `provision.sh` + relay + Guacamole upsert + perm grant. Installed on m2 (`~/m2o/desktop/launch-desktop.sh`) and m2.2 (`~/machinemachine-core/m2o/desktop/launch-desktop.sh`).
+
+### Fixed
+- `desktop/provision.sh` (both hosts) — Traefik router `Host()` was hardcoded to `m2o.machinemachine.ai`, breaking `/console/<name>` for every m2.2-hosted desktop (m2o.machinemachine.ai points at m2, not m2.2). Now reads `CONSOLE_PUBLIC_HOST` from `console-auth/secrets.env` (m2.2 has it set to `console.m-2.cc`; m2 defaults to `m2o.machinemachine.ai`). Also patched the existing `console-euroclean.yaml` in place.
+- `scripts/launch-m2o-desktop.sh` — removed the `docker restart guacamole-full` step from step 5. Guacamole picks up DB changes on next login/session; the restart invalidated every active user's session cookie and made connections briefly appear to "disappear" from the UI.
+
+### Deployed (side-effects of this session)
+- **m2o-console-link Guacamole extension** — the `Console ↗` button on each Guacamole home-screen connection row (source: `~/m2o/guacamole-ext/console-link/` on m2). Extension was already fully written; wasn't installed. Built + deployed via `install-console-link.sh`.
+- **`CONSOLE_HOST_OVERRIDES` env in console-auth** — new optional env, JSON `{slug: host}`. `_host_for(desktop)` returns the override or falls back to `PUBLIC_HOST`. Needed so `/issue` and `/issue-web` return `console.m-2.cc` URLs for m2.2 desktops. Patched into `~/m2o/console-auth/app.py` on m2; container rebuilt + redeployed. `launch-desktop.sh` now auto-updates this map when provisioning a new m2.2 desktop.
+- **§12b subsection: Provisioning the m2-gpt tenant / agent / bearer** — every desktop's Hermes needs its own bearer (not the fleet-wide placeholder from `~/.m2-gpt-key`) to be a real fleet citizen. Documents the one-command `m2gw-provision-agent.sh` flow and the default route chain (spark-glm → deepseek-spark).
+- `scripts/m2gw-provision-agent.sh` — new idempotent one-command wrapper: creates the tenant + agent (with configurable route chain), mints a fresh bearer, and (optionally) patches the target desktop's Hermes `config.yaml` and restarts `hermes-gateway`. Installed on m2 (`~/m2-gpt/m2-gpt/scripts/` **and** `~/m2o/desktop/`) and m2.2 (`~/machinemachine-core/m2o/desktop/`).
+- `scripts/launch-m2o-desktop.sh` — new `--provision-m2gw` flag (+ `--m2gw-primary`, `--m2gw-fallback`, `--m2gw-model`, `--m2gw-principal`) makes the desktop spawn + gateway wire a single command.
+
+### Bring-up done this session
+- **euroclean tenant + agent + bearer** on `gpt.machinemachine.ai`, route chain `spark-glm → deepseek-spark`, Hermes on `euroclean-m2o` wired to the per-agent bearer with `model.default: m2gw-spark-glm/glm-5.3-flash`. End-to-end smoke passed (gateway → chain → LLM → response).
+
+### Authors
+- Mariusz (operator) — hit the pain, asked for the doc + script
+- Fable — end-to-end euroclean provisioning + doc/script extraction
+
 ## [0.2.0] — 2026-02-19
 ### Added
 - Section 10: Orchestrator Pattern — conductor/player separation, structured spawn handoff, when to break the rule

diff --git a/PLAYBOOK.md b/PLAYBOOK.md
@@ -617,6 +617,226 @@ Each cycle, the fleet gets more capable. This is not metaphor — it is the mech
 
 ---
 
+## 12b. m2o Desktop Provisioning (RDP + cross-host Guacamole)
+
+Every fleet agent runs on an **m2o desktop** — a persistent Ubuntu container with `x11vnc`, `xrdp`, `guacd`, `ttyd`, and Hermes baked in (primus image). Two hosts run desktops today:
+
+| Host | LAN IP(s) | SSH user | provision.sh path | Local Guacamole? |
+|------|-----------|----------|-------------------|------------------|
+| **m2** (primary) | `192.168.31.224`, `192.168.31.28` | `m2` | `~/m2o/desktop/provision.sh` | ✅ `guacamole-full` + `guacamole-db` (Coolify service `e0o8o8cowkswcwsgs4so48s8`) |
+| **m2.2** (secondary) | `192.168.31.34` | `m2.2` | `~/machinemachine-core/m2o/desktop/provision.sh` | ❌ own Coolify, no Guacamole stack |
+
+> **Path gotcha:** on m2.2 the script is under `~/machinemachine-core/m2o/`, not `~/m2o/`. The `m2o-provision` skill doc uses the upstream `~/m2o/` path — treat as advisory, not literal for m2.2.
+
+> **DNS gotcha:** from inside fleet docker containers (e.g. anything on the `coolify` network), both `m2` and `m2.2` frequently resolve to the *same* tailnet address (whichever host the container runs on). When you need to reach the *other* host, use its LAN IP, not the name.
+
+### Spawn a desktop — one command, full setup
+
+Use the **`launch-desktop.sh`** wrapper (installed at `~/m2o/desktop/launch-desktop.sh` on m2 and `~/machinemachine-core/m2o/desktop/launch-desktop.sh` on m2.2; canonical source: `scripts/launch-m2o-desktop.sh` in this repo). It runs `provision.sh`, waits for healthy, creates the socat guacd relay (m2.2 only), upserts the Guacamole RDP row on m2, and grants perms — all idempotent.
+
+```bash
+# on the target host, as its own user (m2 or m2.2)
+cd ~/m2o/desktop      # or ~/machinemachine-core/m2o/desktop on m2.2
+./launch-desktop.sh <name> [--vnc-pass X] [--grant-users guacadmin,m2,...]
+```
+
+Flags: `--relay-port <n>` (default: auto-pick next `X4822`), `--m2-ssh <user@host>` (default `m2@192.168.31.224`), `--skip-guacamole`, `--force-recreate`.
+
+**Raw provision (if you only want the container, no Guacamole wiring):**
+
+```bash
+cd ~/m2o/desktop      # or ~/machinemachine-core/m2o/desktop on m2.2
+./provision.sh <name> [vnc_password]     # e.g. ./provision.sh euroclean
+```
+
+Creates container `<name>-m2o` on the `coolify` docker network + named volumes `<name>-{agent-home,workspace,m2home}` (survive re-provision). `M2_GPT_API_KEY` is injected from `~/.m2-gpt-key` (host side, per-user). Fleet standard 2026-07-09: **RDP is the default access path**; VNC is fallback.
+
+### First-boot wait (~2 min "unhealthy" is normal)
+
+The entrypoint does a long `rm -rf /home` before x11vnc/xrdp start. The container will show `unhealthy` and ports 5900/3389 will refuse connections for 60–120s. Don't kill it — the health check flips once services come up. To watch:
+
+```bash
+docker inspect <name>-m2o --format '{{.State.Health.Status}}'
+docker exec <name>-m2o bash -lc 'ps -eo pid,etime,cmd | grep -E "rm -rf|x11vnc|xrdp|guacd" | grep -v grep'
+```
+
+### Expected WARNs on m2.2
+
+Because m2.2 doesn't yet run its own Guacamole and only bridges console-auth via socat:
+
+- `WARN: guacamole-db container not found — create the connection manually` — expected. Wire via `launch-desktop.sh` or the SQL in the next subsection.
+- `WARN console: console-auth:latest image missing — cred saved to secrets.env` — expected. m2.2 doesn't host the console-auth image; a socat container named `console-auth` on m2.2's `coolify` network forwards `:8080` to `192.168.31.224:28080` (m2's `console-auth-lan-relay`), so Traefik's `forwardAuth` middleware works transparently. The cred **does still need to reach m2**: append the entry from `~/machinemachine-core/m2o/console-auth/secrets.env` (the `"<name>":"admin:..."` line inside `CONSOLE_BASIC_CREDS`) into m2's `~/m2o/console-auth/secrets.env` and restart `console-auth` on m2. See [Console (ttyd)](#console-ttyd-magic-link-flow) below.
+
+### Console (ttyd, magic-link flow)
+
+The `/console/<desktop>` browser terminal is served by ttyd inside the desktop container, fronted by Traefik with a `forwardAuth` middleware pointing at `console-auth`. There is only **one real `console-auth`** in the fleet — it runs on m2 (FastAPI, `console-auth:latest`, source at `~/m2o/console-auth/`). Both hosts route to it:
+
+- **m2 desktops** → m2's Traefik → `console-auth` (same docker network) → ttyd inside the desktop container. Public host: `m2o.machinemachine.ai`.
+- **m2.2 desktops** → m2.2's Traefik (public host: `console.m-2.cc`, Cloudflare) → m2.2's socat `console-auth` → m2's `console-auth-lan-relay:28080` → real console-auth on m2 → response back up the chain. ttyd is inside the desktop container on m2.2.
+
+**Two things must be in sync per desktop** for the browser console to work:
+
+1. **Traefik router `Host()`** must match the desktop's host (`m2o.machinemachine.ai` on m2, `console.m-2.cc` on m2.2). Old `provision.sh` hardcoded `m2o.machinemachine.ai` for both — **patched 2026-08-24** to read `CONSOLE_PUBLIC_HOST` from `console-auth/secrets.env` (m2.2's secrets sets it to `console.m-2.cc`; m2 leaves the default).
+2. **The cred entry** (auto-generated per desktop by provision.sh into the local `secrets.env`) must exist in m2's `CONSOLE_BASIC_CREDS`. On m2 that's automatic (provision.sh redeploys the local console-auth). On m2.2 you must manually copy the `"<name>":"admin:..."` entry into m2's `~/m2o/console-auth/secrets.env` and `docker rm -f console-auth && docker run -d --name console-auth --restart unless-stopped --network coolify --env-file secrets.env console-auth:latest && docker network connect e0o8o8cowkswcwsgs4so48s8 console-auth`.
+
+### Guacamole "Console ↗" button (m2o-console-link extension)
+
+A tiny Guacamole extension (`~/m2o/guacamole-ext/console-link/` on m2) adds a `Console ↗` button next to each connection on the Guacamole home page. Click → POST to `/console-mint/issue-web` with the current Guacamole session token → console-auth validates the caller against the Guacamole REST API, mints a magic link for that desktop, and the browser opens the URL in a new tab.
+
+- **Traefik route:** `/data/coolify/proxy/dynamic/console-mint.yaml` on m2 exposes `console-auth` at `Host(m2o.machinemachine.ai) && PathPrefix(/console-mint)` with a `stripPrefix` middleware. Same origin as Guacamole, so no CORS.
+- **Which rows get a button:** the extension calls `/console-mint/consoles` to get the list of desktop slugs; rows whose name slugifies onto a known desktop show the button. Everything else is hidden.
+- **Per-desktop host in the returned URL:** by default `console-auth` returns `https://<CONSOLE_PUBLIC_HOST>/console/<name>/?t=<token>`. For m2.2-hosted desktops, the URL must instead be on `console.m-2.cc`. This is handled by the `CONSOLE_HOST_OVERRIDES` env in `~/m2o/console-auth/secrets.env` on m2 — a JSON dict `{slug: host}`. `launch-desktop.sh` on m2.2 auto-adds the new desktop to this map and redeploys `console-auth` on m2.
+- **Install / update the extension:** on m2, `bash ~/m2o/guacamole-ext/install-console-link.sh`. Rebuilds the JAR, stages it at `/data/coolify/guacamole-home/extensions/` (for the permanent GUACAMOLE_HOME bind mount), copies it into the live container, and reloads the webapp by `touch`ing `web.xml` — no container restart, no session invalidation.
+- **Why not a container restart:** the Guacamole entrypoint runs `rm -Rf $HOME/.guacamole` on every container start, which wipes JARs dropped into `~/.guacamole/extensions/`. The bind-mount + `touch web.xml` pattern is the durable install path.
+
+### Provisioning the m2-gpt tenant / agent / bearer
+
+Every desktop's Hermes talks to the fleet LLM gateway at `https://gpt.machinemachine.ai/v1`. It authenticates with a **per-agent** bearer (`sk-m2-...`), argon2-hashed at rest — not the fleet-wide `~/.m2-gpt-key` that `provision.sh` bakes in as a placeholder. Without a real per-agent bearer, the desktop is off-fleet: it shares tenancy, budget, and memory with `m2` itself.
+
+**One command** (installed alongside `launch-desktop.sh` on both hosts, canonical source: `scripts/m2gw-provision-agent.sh` in this repo):
+
+```bash
+# on m2 (or from m2.2 — script SSHes to m2 for gateway ops)
+m2gw-provision-agent.sh <slug>                                        # sensible defaults
+m2gw-provision-agent.sh <slug> --primary-route <id> \
+                               --fallback-route <id> \
+                               --default-model <name> \
+                               --principal <email> \
+                               --hermes-container <container>          # patches Hermes config
+```
+
+Idempotent: existing tenant is reused; existing agent has its `route_bindings` updated; **a fresh bearer is minted every run** (old ones stay `active` — revoke by hand if you want single-key hygiene). If `--hermes-container` matches a container running on this host, the script patches `/home/developer/.hermes/config.yaml` (`model.default` + `model.api_key`) and restarts `hermes-gateway`.
+
+**Defaults** (chosen 2026-08-28 for the euroclean bring-up):
+
+| Field | Value |
+|-------|-------|
+| Primary route (priority 1) | `spark-glm` — self-hosted spark cluster, `glm-5.3-flash` (multimodal, vision-capable) |
+| Fallback route (priority 2) | `deepseek-spark` — self-hosted spark, `deepseek-v4-flash-0731` (text) |
+| Hermes `model.default` | `m2gw-spark-glm/glm-5.3-flash` |
+| Tenant budget | $20/mo |
+| Retention | 90 days |
+
+> Note on model naming: the operator brief asked for `deepseek-v4-3107`. That model id isn't currently deployed on the fleet — the spark cluster serves `deepseek-v4-flash-0731`. Either rename the target model on the upstream, or add a new route entry in `routes` (see `scripts/set-default-route-chain.sh` in the m2-gpt repo for the pattern).
+
+**All at once via `launch-desktop.sh`:**
+
+```bash
+./launch-desktop.sh <name> --provision-m2gw \
+   [--m2gw-primary spark-glm] \
+   [--m2gw-fallback deepseek-spark] \
+   [--m2gw-model m2gw-spark-glm/glm-5.3-flash] \
+   [--m2gw-principal <email>]
+```
+
+Runs the same wrapper as step 6 after the desktop is healthy. The bearer is printed once — save it before it scrolls off.
+
+**Route inventory (read-only glance) — introspect the gateway DB:**
+
+```bash
+# on m2, inside the prod gateway container
+PGW=$(docker ps --format '{{.Names}}' | grep '^gateway-akvnse3p7' | head -1)
+docker exec $PGW python3 -c "
+import os, asyncio, asyncpg
+async def m():
+    URL=os.environ['M2GW_DATABASE_URL'].replace('postgresql+asyncpg://','postgresql://')
+    c=await asyncpg.connect(URL)
+    for r in await c.fetch('select id, provider, base_url, default_params->>' + chr(39) + 'model' + chr(39) + ' as model, usage_scope from routes order by id'):
+        print(dict(r))
+    await c.close()
+asyncio.run(m())
+"
+```
+
+Existing routes at the time of writing: `spark-glm` (glm-5.3-flash, spark cluster), `deepseek-spark` (deepseek-v4-flash-0731, spark), `glm-5.2` / `glm-5.2-zai` / `glm-5.3-zai` (Z.ai proxy), `qwen-fast` / `qwen-local` / `ornith-coding` (local coding models).
+
+### Mint a magic link (admin key, from anywhere on m2)
+
+```bash
+source ~/m2o/console-auth/secrets.env
+CA_IP=$(docker inspect console-auth --format '{{(index .NetworkSettings.Networks "coolify").IPAddress}}')
+curl -s -X POST -H "X-Api-Key: $CONSOLE_AUTH_ADMIN_KEY" \
+     -H 'Content-Type: application/json' \
+     -d '{"desktop":"<name>"}' \
+     http://$CA_IP:8080/issue
+# → returns {"url":"https://m2o.machinemachine.ai/console/<name>/?t=..."}
+# For m2.2 desktops, swap the host to console.m-2.cc — the token is valid on any host
+# (verify uses x-forwarded-host). Link is single-use, 15 min; session cookie is 24 h.
+```
+
+### Wiring an m2.2 desktop into m2's Guacamole
+
+m2's `guacamole-full` needs to speak the guacd protocol to the desktop container (port 4822). On m2.2 that port lives inside the container on the `coolify` docker network — not published to the host. The established pattern is a **per-desktop `alpine/socat` relay** on m2.2 that publishes a unique host port and forwards to the desktop's internal `guacd:4822`.
+
+Existing port allocations (append your own):
+
+| Desktop | Host port on m2.2 |
+|---------|-------------------|
+| dealflow-legacy | 14822 |
+| dealflow | 24823 |
+| m2o-operator | 24822 |
+| euroclean | 34822 |
+
+**Step 1 — publish guacd on m2.2:**
+
+```bash
+# on m2.2, pick an unused port
+docker run -d --name <name>-guacd-relay --restart unless-stopped \
+  --network coolify -p <PORT>:<PORT> \
+  alpine/socat tcp-listen:<PORT>,fork,reuseaddr tcp:<name>-m2o:4822
+```
+
+**Step 2 — insert Guacamole connection on m2** (via `guacamole-db`, user `root` / `guacamole_root_pass`):
+
+```sql
+INSERT INTO guacamole_connection (connection_name, protocol, proxy_hostname, proxy_port)
+VALUES ('<Name> Desktop', 'rdp', '192.168.31.34', <PORT>);
+SET @id = LAST_INSERT_ID();
+
+INSERT INTO guacamole_connection_parameter (connection_id, parameter_name, parameter_value) VALUES
+  (@id,'hostname','127.0.0.1'), (@id,'port','3389'),
+  (@id,'username','developer'), (@id,'password','<vnc_password>'),
+  (@id,'width','1920'), (@id,'height','1080'), (@id,'color-depth','32'),
+  (@id,'security','any'), (@id,'ignore-cert','true'),
+  (@id,'enable-drive','true'), (@id,'drive-name','Shared'),
+  (@id,'drive-path','/home/developer/Desktop/Shared'),
+  (@id,'create-drive-path','true'), (@id,'enable-sftp','false');
+
+INSERT INTO guacamole_connection_permission (entity_id, connection_id, permission)
+SELECT e.entity_id, @id, p.perm
+FROM guacamole_entity e
+CROSS JOIN (SELECT 'READ' AS perm UNION ALL SELECT 'UPDATE'
+            UNION ALL SELECT 'DELETE' UNION ALL SELECT 'ADMINISTER') p
+WHERE e.name = 'guacadmin' AND e.type = 'USER';
+```
+
+**Critical schema gotchas** (inherited from m2's Guacamole):
+
+- `proxy_hostname` / `proxy_port` live on the `guacamole_connection` table, **not** on `guacamole_connection_parameter`.
+- `hostname` inside the connection parameters stays `127.0.0.1` — guacd runs *inside* the desktop container, right next to x11vnc/xrdp.
+- Use the `root` mysql user for writes. `guacamole_user` is SELECT-only.
+
+**Naming standard 2026-07-09:** plain `'<Name> Desktop'` = RDP (default). Append `' (VNC)'` only for the fallback VNC row if you also create one.
+
+### Access without Guacamole (m2.2 direct)
+
+```bash
+IP=$(docker inspect <name>-m2o --format '{{(index .NetworkSettings.Networks "coolify").IPAddress}}')
+# from m2.2:            RDP client → $IP:3389   (developer / <vnc_password>)
+# from a workstation:   ssh -L 3389:$IP:3389 m2.2@192.168.31.34, then RDP to localhost:3389
+```
+
+### Managing
+
+```bash
+docker ps --filter name=-m2o                          # list all m2o desktops on this host
+docker rm -f <name>-m2o && ./provision.sh <name>      # rebuild (volumes preserved)
+docker volume ls | grep <name>-                       # named volumes
+docker inspect <name>-m2o --format '{{.State.Health.Status}}'
+```
+
+---
+
 ## 13. Infrastructure Architecture
 
 The full infrastructure spec lives in: **[sections/architecture.md](sections/architecture.md)**