feat: bundle a stateless MCP server into the server at /mcp with OAuth 2.1

.changeset/bundled-mcp-server.md

@@ -0,0 +1,11 @@
+---
+'@devicesdk/server': minor
+'@devicesdk/cli': minor
+'@devicesdk/website': patch
+---
+
+Add a stateless Streamable-HTTP MCP server bundled into `@devicesdk/server` at `/mcp` - 15 tools covering projects, devices, env vars, script versions, commands, and offline docs search, authenticated via OAuth 2.1 (PKCE + dynamic client registration) or existing API tokens. Every tool re-enters the REST API in-process so behavior always matches the server's own version.
+
+This replaces the standalone `@devicesdk/mcp` npm package, which is removed - `devicesdk init` now scaffolds `.mcp.json` pointing at the server's own `/mcp` endpoint instead of `npx @devicesdk/mcp`. The docs site's MCP page is rewritten to match.
+
+`GET /v1/projects/:projectId/devices/:deviceId/logs` is un-deprecated: it was a permanent 410 in the Cloudflare era to protect a Durable Object rows-read quota that no longer exists on the self-hosted SQLite server, so it now returns a real cursor-paginated page of persisted logs. This is what makes `devicesdk_device_logs` a working MCP tool instead of a stub. The watcher WebSocket (`/watch?backfillLimit=N`) remains the dashboard's live-tailing path and is unchanged.

.dockerignore

@@ -15,6 +15,5 @@ firmware/esp32/build
 firmware/esp32/managed_components
 firmware/pico/build
 apps/website
-docs
 data
 *.log

AGENTS.md

@@ -54,13 +54,14 @@ Docker): `devicesdk.sqlite` (WAL),
 
 ## Monorepo Architecture
 
-pnpm + Turborepo. Bun is the **server runtime only** - the CLI/MCP run on plain
+pnpm + Turborepo. Bun is the **server runtime only** - the CLI runs on plain
 Node for npm users.
 
 **`apps/server`** (`@devicesdk/server`) - THE backend. Bun + Hono + Chanfana
 (OpenAPI) + Zod + bun:sqlite. One process, one port (8080): REST API under
 `/v1/*`, device + watcher WebSockets, dashboard SPA static serving, OpenAPI docs
-at `/api-docs`.
+at `/api-docs`, and a bundled MCP server at `/mcp` (OAuth 2.1 or static API
+tokens) for AI coding agents.
 
 **`apps/dashboard`** - Vue 3 + Quasar SPA. Local email/password auth
 (register/login). Built `dist/spa` is served same-origin by the server;
@@ -76,8 +77,6 @@ in `~/.devicesdk/credentials.json` → mDNS auto-discovery (`devicesdk.local`).
 non-default setups or when mDNS is unavailable. `devicesdk dev` still uses the
 workerd simulator (convergence on the server runtime is a roadmap item).
 
-**`packages/mcp`** - MCP server wrapping the CLI for AI agents.
-
 **`apps/simulation`** - Vue UI for the CLI dev simulator (built dist embedded in
 CLI).
 
@@ -139,8 +138,27 @@ image.
 - **Metrics**: `src/foundation/usageMetrics.ts` - 5-minute SQLite buckets in
   `device_usage`; windows 1h/12h/7d. No cost estimation (that was a
   cloud-billing concept).
-- **Janitor**: `src/janitor.ts` hourly - expired sessions/CLI codes, old
-  logs/usage.
+- **MCP**: `src/mcp/` - stateless Streamable HTTP MCP server at `POST /mcp`
+  (`@modelcontextprotocol/sdk` + `@hono/mcp`, `sessionIdGenerator: undefined`,
+  `enableJsonResponse: true` - a fresh `McpServer` per request, no
+  `Mcp-Session-Id`, no server-to-client push). Every `devicesdk_*` tool
+  re-enters the REST API in-process (`app.request(path, init, c.env)`,
+  forwarding the caller's own Authorization/Cookie header) instead of
+  duplicating query logic - one source of truth for validation and response
+  shapes. `devicesdk_docs_search` queries a SQLite FTS5 index built at server
+  build time from `docs/public/**/*.md` (`scripts/build-docs-index.ts`), not
+  a network call. GET/DELETE /mcp are 405 (stateless - nothing to resume or
+  terminate).
+- **OAuth**: `src/oauth/` - a minimal OAuth 2.1 authorization server
+  additive to static API tokens: PKCE-required authorization-code grant, open
+  (rate-limited) dynamic client registration (RFC 7591), no refresh tokens.
+  Access tokens are `tokens` rows with `managed=1` and a real `expires_at`
+  (30 days) - revocable from the dashboard's Tokens page like any other
+  token. `foundation/auth.ts`'s `mcpAuth` wrapper adds a `WWW-Authenticate`
+  header on 401s so MCP clients discover
+  `/.well-known/oauth-protected-resource` automatically.
+- **Janitor**: `src/janitor.ts` hourly - expired sessions/CLI codes, OAuth
+  codes, expired API tokens, old logs/usage.
 - **mDNS**: `src/foundation/mdns/` - a zero-dependency multicast-DNS responder
   (`node:dgram`) advertising the server as `<MDNS_HOSTNAME>.local` (default
   `devicesdk`) so LAN devices resolve it without a static IP. `dnsPacket.ts` is
@@ -177,6 +195,9 @@ on the user's own server - that's the trust model).
 | Usage metrics | `apps/server/src/foundation/usageMetrics.ts` |
 | Watch WebSocket routes | `apps/server/src/endpoints/devices/wsRoutes.ts` |
 | Dashboard watch composable | `apps/dashboard/src/composables/useDeviceStream.ts` |
+| Bundled MCP server (`/mcp`) | `apps/server/src/mcp/` |
+| OAuth 2.1 authorization server | `apps/server/src/oauth/` |
+| Docs FTS5 index builder | `apps/server/scripts/build-docs-index.ts` |
 | HA entity types/persistence | `packages/core` + `apps/server/src/endpoints/devices/{get,upsert}DeviceEntities.ts` |
 | ESP32/Pico image checksum patching | `apps/server/src/foundation/{esp32ImageChecksum,picoUf2Checksum}.ts` |
 | Endpoint patterns | `apps/server/src/endpoints/` (Hono + Chanfana + Zod) |
@@ -230,7 +251,7 @@ on the user's own server - that's the trust model).
 
 - **Before every commit**, run `pnpm lint`. Do not commit if linting fails.
 - **Every PR must include a changeset** referencing every workspace package
-  touched (npm-published: `@devicesdk/core`, `@devicesdk/cli`, `@devicesdk/mcp`;
+  touched (npm-published: `@devicesdk/core`, `@devicesdk/cli`;
   private-with-changelog: `@devicesdk/server`, `@devicesdk/dashboard`,
   `@devicesdk/simulation`, `@devicesdk/website`). Create it early in the branch
   with `pnpm changeset` so CI can validate it.
@@ -253,7 +274,7 @@ on the user's own server - that's the trust model).
 The monorepo uses `@changesets/cli` for versioning and release management.
 
 - **Public packages** (`private: false`) are published to npm by `pnpm release`:
-  `@devicesdk/core`, `@devicesdk/cli`, `@devicesdk/mcp`.
+  `@devicesdk/core`, `@devicesdk/cli`.
 - **Private packages** (`private: true`) are version-bumped and get changelog
   entries, but are **not** published to npm. This includes the runtime apps:
   `@devicesdk/server`, `@devicesdk/dashboard`, `@devicesdk/simulation`, and

CONTRIBUTING.md

@@ -29,13 +29,12 @@ This is a pnpm + Turborepo monorepo:
 
 | Path | Package | Purpose |
 |---|---|---|
-| `apps/server` | `@devicesdk/server` | Bun backend: REST API, device WebSockets, dashboard hosting |
+| `apps/server` | `@devicesdk/server` | Bun backend: REST API, device WebSockets, dashboard hosting, bundled MCP server (`/mcp`) |
 | `apps/dashboard` | `@devicesdk/dashboard` | Vue 3 + Quasar dashboard SPA |
 | `apps/website` | `@devicesdk/website` | Vue 3 + Vite SSG marketing and docs site |
 | `apps/simulation` | `@devicesdk/simulation` | CLI dev simulator UI |
 | `packages/core` | `@devicesdk/core` | Shared types and `DeviceEntrypoint` base class (published to npm) |
 | `packages/cli` | `@devicesdk/cli` | `devicesdk` CLI (published to npm) |
-| `packages/mcp` | `@devicesdk/mcp` | Model Context Protocol server (published to npm) |
 | `packages/typescript-config` | `@repo/typescript-config` | Shared tsconfig base |
 | `firmware/esp32`, `firmware/pico` | - | ESP32 / Pico W firmware |
 | `examples/*` | - | Example device projects |

Dockerfile

@@ -26,6 +26,12 @@ WORKDIR /repo
 COPY --from=build /repo /repo
 RUN cd apps/server \
 	&& bun build src/server.ts --target=bun --outfile /out/server.js
+# Build-time SQLite FTS5 index of docs/public/**/*.md for devicesdk_docs_search
+# (offline, version-pinned to this image - see apps/server/scripts/build-docs-index.ts).
+# Docs reach the build context via stage 1's `COPY . .` (.dockerignore does not
+# exclude docs/, since apps/server's docs-index build needs docs/public).
+RUN cd apps/server \
+	&& bun run scripts/build-docs-index.ts ../../docs/public /out/docs-index.sqlite
 
 # ---- stage 2.5: fetch prebuilt firmware binaries (best-effort) ----
 # Firmware workflows publish versioned releases (tags firmware-esp32@vX.Y.Z /
@@ -65,6 +71,7 @@ RUN mkdir -p /firmwares && cd /firmwares \
 FROM oven/bun:1.3.14-slim
 WORKDIR /app
 COPY --from=serverbuild /out/server.js /app/server.js
+COPY --from=serverbuild /out/docs-index.sqlite /app/docs-index.sqlite
 COPY --from=build /repo/apps/server/migrations /app/migrations
 COPY --from=build /repo/apps/dashboard/dist/spa /app/public
 COPY --from=firmware /firmwares /app/firmwares-dist
@@ -73,7 +80,8 @@ ENV PORT=8080 \
 	DATA_DIR=/data \
 	PUBLIC_DIR=/app/public \
 	MIGRATIONS_DIR=/app/migrations \
-	FIRMWARES_DIST_DIR=/app/firmwares-dist
+	FIRMWARES_DIST_DIR=/app/firmwares-dist \
+	DOCS_INDEX_PATH=/app/docs-index.sqlite
 
 # Run as an unprivileged user. The bun image ships a `bun` group/user (uid 1000),
 # so reuse it rather than creating a new account that may collide with the host.

NOTICE

@@ -87,6 +87,6 @@ categories of licenses in the dependency tree are:
 - ISC
 
 No GPL or AGPL dependencies are used in the published npm packages
-(@devicesdk/core, @devicesdk/cli, @devicesdk/mcp). The server application
+(@devicesdk/core, @devicesdk/cli). The server application
 (apps/server) is licensed under AGPL-3.0 and may include AGPL-compatible
 dependencies.

README.md

@@ -89,17 +89,16 @@ docker build -t devicesdk .
 
 ## Project Structure
 
-pnpm + Turborepo monorepo. **Bun is the server runtime only**: the CLI and MCP run on plain Node for npm users.
+pnpm + Turborepo monorepo. **Bun is the server runtime only**: the CLI runs on plain Node for npm users.
 
 | Package | Name | Description |
 |---|---|---|
-| `apps/server` | `@devicesdk/server` | The backend: Bun + Hono + Chanfana + Zod + `bun:sqlite`. One process, one port: REST API (`/v1/*`), device + watcher WebSockets, dashboard SPA, OpenAPI docs (`/api-docs`) |
+| `apps/server` | `@devicesdk/server` | The backend: Bun + Hono + Chanfana + Zod + `bun:sqlite`. One process, one port: REST API (`/v1/*`), device + watcher WebSockets, dashboard SPA, OpenAPI docs (`/api-docs`), and a bundled MCP server (`/mcp`) for AI coding agents |
 | `apps/dashboard` | `@devicesdk/dashboard` | Vue 3 + Quasar SPA: local email/password auth, project/device/token management. Served same-origin by the server |
 | `apps/simulation` | `@devicesdk/simulation` | Vue 3 device-simulation UI (static export consumed by the CLI `dev` command) |
 | `apps/website` | `@devicesdk/website` | Vue 3 + Vite SSG marketing & docs site |
 | `packages/core` | `@devicesdk/core` | Shared TypeScript types and the `DeviceEntrypoint` base class (published to npm) |
 | `packages/cli` | `@devicesdk/cli` | CLI tool (`devicesdk`): login, init, build, dev, deploy, flash, logs, status, inspect |
-| `packages/mcp` | `@devicesdk/mcp` | Model Context Protocol server wrapping the CLI for AI agents |
 | `packages/typescript-config` | `@repo/typescript-config` | Shared tsconfig base |
 | `firmware/esp32` | | ESP32 firmware (ESP-IDF, WebSocket client) |
 | `firmware/pico` | | Raspberry Pi Pico W firmware (C++, lwIP WebSocket client) |

TROUBLESHOOT.md

@@ -1,5 +1,19 @@
 # Troubleshooting Log
 
+### `docker build` fails on a docs-consuming build step with "directory not found", even though the Dockerfile does `COPY . .`
+**Date**: 2026-07-08
+**Question/Problem**: Adding a `RUN` step to the `serverbuild` stage that reads `docs/public/**/*.md` (`apps/server/scripts/build-docs-index.ts`, for the `/mcp` docs-search FTS5 index) failed with `Docs directory not found: /repo/docs/public`, even though stage 1 does `COPY . .` into `/repo` and stage 2 does `COPY --from=build /repo /repo`.
+**Root Cause**: `.dockerignore` had a bare `docs` line. `.dockerignore` exclusions apply to the **build context** sent to the daemon before any stage runs - they are not a per-`COPY` filter, so no later stage can retrieve a path `.dockerignore` excluded, no matter how many `COPY --from=` hops happen afterward. The line was originally added because only `apps/website` consumed `docs/public` (and `apps/website` itself is `.dockerignore`d, since the Docker image doesn't build the website); once `apps/server`'s build gained its own reason to read `docs/public`, the blanket exclusion silently broke it.
+**Solution**: Removed the bare `docs` line from `.dockerignore`. `docs/public` is ~60 small Markdown files (under 400K) with no secrets, so including it in the build context is cheap and safe - the runtime image still only ships the *compiled* `docs-index.sqlite` (verified with `find / -xdev -name '*.md'` inside a running container: nothing from `docs/public` leaks in, only the pre-existing unrelated `apps/server/migrations/README.md`).
+**Rule**: When a Dockerfile step reads a path under `COPY . .`, check `.dockerignore` for that path *first* if the read fails - grep it before adding debug `RUN ls` steps. A blanket top-level directory exclusion (`docs`, not `apps/website/docs` or similar) silently forecloses every future consumer of that directory, not just the one it was written for.
+
+### `pnpm test --filter @devicesdk/server` (`bun test src`) silently skips `tests/e2e/` and `tests/unit/` - a green run does not mean those suites passed
+**Date**: 2026-07-08
+**Question/Problem**: After adding `apps/server/tests/e2e/mcp.test.ts`, `apps/server/tests/e2e/oauth.test.ts`, and `apps/server/tests/unit/docs-index.test.ts`, running the command CLAUDE.md lists as "server unit tests" (`pnpm test --filter @devicesdk/server`) reported a normal-looking pass with a *lower* test count than expected - it never ran the new files at all, and gave no error or warning that it hadn't.
+**Root Cause**: `apps/server/package.json` has three different test scripts with different scopes: `"test": "bun test src"`, `"test:e2e": "bun test src tests"`, `"test:coverage": "bun run scripts/coverage-gate.ts"` (which internally also runs `bun test src tests --coverage`). `bun test <patterns>` filters test files by a **substring match on file path** - `bun test src` only ever discovers files whose path contains `src`, so anything under `apps/server/tests/` (e2e and unit alike) is never loaded, and bun does not warn that a pattern matched fewer files than exist. The root `CLAUDE.md`'s command table lists only `pnpm test --filter @devicesdk/server` as "server unit tests", which reads as the full gate but isn't.
+**Solution**: For a true full-suite run (e.g. before reporting a task complete, or in CI-equivalent verification), run `bun test src tests` (or `pnpm test:e2e --filter @devicesdk/server` / `pnpm test:coverage --filter @devicesdk/server`) directly from `apps/server`, not the bare `test` script.
+**Rule**: In this repo, `apps/server`'s plain `test` script is a subset (`src/` colocated tests only). Anything under `apps/server/tests/` (e2e, unit) needs `test:e2e` or `test:coverage` to run at all - never trust a green `pnpm test --filter @devicesdk/server` alone as evidence that e2e/unit tests pass; confirm the file count in the summary line matches what you expect, or run the fuller command explicitly.
+
 ### Dashboard won't load: assets served with empty MIME type (cors drops `Bun.file` Content-Type)
 **Date**: 2026-06-29
 **Question/Problem**: Opening the self-hosted dashboard (`docker compose up`, `http://localhost:8080`) shows a blank page. Console: `Refused to apply style from '.../assets/index-*.css' because its MIME type ('') is not a supported stylesheet MIME type` and `Failed to load module script: ... responded with a MIME type of ""`. The `/assets/*.css` and `*.js` responses have an **empty `Content-Type`**.
@@ -352,10 +366,11 @@ uint8_t com_pins = (height == 32) ? 0x02 : 0x12;
 **Rule**: For any new SSD1306-family panel that isn't the canonical 128×32, expect alternating COM pins. The clue that you have the wrong config is diagnostic patterns that show up as every-other-row stripes, not as rectangles.
 
 ### `devicesdk logs --tail` exits / fails behind a corporate proxy
-**Date**: 2026-05-01
+**Date**: 2026-05-01 (superseded 2026-07-08, see below)
 **Question/Problem**: Since May 2026 the CLI `logs` and `logs --tail` commands open a WebSocket to `/v1/projects/.../devices/.../watch` instead of polling the deprecated `/logs` HTTP endpoint. Some corporate proxies strip the `Upgrade: websocket` header or block 101 responses, leaving the CLI unable to connect.
-**Root Cause**: The deprecation removed the HTTP polling fallback. The watcher endpoint is the only path that delivers logs.
-**Solution**: Open your self-hosted dashboard at `http(s)://<your-server>/projects/<slug>/devices/<slug>` - the logs panel uses the same WebSocket and works in any browser the user can already reach. If neither WS nor the dashboard is reachable, raise the issue with the network operator (the watcher socket is required for the runtime UI). Do **not** reintroduce a polling fallback - the burn pattern that triggered the migration would recur.
+**Root Cause**: At the time, the `/logs` HTTP endpoint had just been made a permanent 410 - it was Cloudflare-era, where every `device_logs` read/write billed against a Durable Object's daily rows-read/rows-written quota, and naive polling clients had burned through it (see the row-budget entry above). The watcher WebSocket was the only path left that delivered logs.
+**Solution (2026-05-01, now historical)**: Open your self-hosted dashboard, which uses the same WebSocket. Do not reintroduce a polling fallback.
+**Update (2026-07-08)**: The self-host refactor replaced the Durable Object with the server's own local SQLite (`device_logs` table) - there is no external rows-read quota to protect anymore, so `GET /v1/projects/:projectId/devices/:deviceId/logs` (cursor/limit/level paged, newest first) is un-deprecated and works normally again; see `apps/server/src/endpoints/logs/listLogs.ts`. It is intentionally **not** wired back into `devicesdk logs --tail` (that command still wants live push, which only the watcher WebSocket provides) - it exists for scripts, cron jobs, and the bundled MCP server's `devicesdk_device_logs` tool, which don't want to hold a socket open. If a corporate proxy blocks the WS, a caller can now poll this endpoint directly as a workaround; the CLI itself is unchanged.
 
 ### Firmware R2 upload fails with "The file … does not exist" in CI
 **Date**: 2026-05-29

apps/dashboard/src/components/DeviceLogs.vue

@@ -84,11 +84,12 @@ const props = defineProps<{
 const $q = useQuasar();
 const levelFilter = ref<string | null>(null);
 
-// Logs are now WS-only. The watcher WebSocket sends up to `backfillLimit`
-// recent entries on connect (replay frames) followed by `history_complete`,
-// then live events. The legacy HTTP `/logs` endpoint returns 410 — see
-// apps/api/src/durableObjects/lib/device.ts `getLogs` for the full incident
-// write-up.
+// This panel stays WS-only for live tailing. The watcher WebSocket sends up
+// to `backfillLimit` recent entries on connect (replay frames) followed by
+// `history_complete`, then live events. A stateless paging GET also exists
+// at /v1/projects/:projectId/devices/:deviceId/logs (see
+// apps/server/src/endpoints/logs/listLogs.ts) for scripts and MCP tools that
+// don't want to hold a socket open, but the dashboard has no need for it.
 const {
   streamedLogs,
   deviceStatus,

apps/dashboard/src/composables/useDeviceStream.ts

@@ -18,8 +18,9 @@ export interface UseDeviceStreamOptions {
  *
  * Auto-reconnects on disconnection with exponential backoff. When
  * `backfillLimit` is provided, replay frames (history) and live events are
- * delivered on the same socket - the dashboard's logs panel uses this to
- * avoid the HTTP `/logs` endpoint, which was deprecated in May 2026.
+ * delivered on the same socket - the dashboard's logs panel uses this single
+ * socket for both history and live tailing instead of paging the HTTP
+ * `/logs` endpoint, which exists but only returns a point-in-time snapshot.
  *
  * Frame format: `{ event, data, replay? }`
  *   - event "status"           → connection state changes