diff --git a/.atlas-analysis.json b/.atlas-analysis.json index 9b25f20..f45d39c 100644 --- a/.atlas-analysis.json +++ b/.atlas-analysis.json @@ -24,8 +24,7 @@ "pages": [ "terminology", "concepts/pki", - "concepts/firmware-signing", - "concepts/deployments" + "concepts/firmware-signing" ] }, { diff --git a/api/websocket-events.mdx b/api/websocket-events.mdx index 53b9b94..348ce42 100644 --- a/api/websocket-events.mdx +++ b/api/websocket-events.mdx @@ -68,7 +68,7 @@ The server is requesting that the device perform a graceful reboot. The payload {} ``` -Upon receiving this event, the device should send a `rebooting` event back to the server (see [Device → Server Events](#device-server-events)), then initiate a system reboot. This event is typically triggered from the NervesHub web console or via the management API. +Upon receiving this event, the device should send a [`rebooting`](#rebooting) event back to the server, then initiate a system reboot. This event is typically triggered from the NervesHub web console or via the management API. *** diff --git a/concepts/deployment-workflows.mdx b/concepts/deployment-workflows.mdx new file mode 100644 index 0000000..99b8950 --- /dev/null +++ b/concepts/deployment-workflows.mdx @@ -0,0 +1,139 @@ +--- +title: "Deployment Group Workflows" +sidebarTitle: "Workflows" +description: "Stage a firmware rollout with a workflow definition: per-step targeting, concurrency, failure tolerance, and approval gates, defined in JSON and uploaded to a deployment group." +--- + +By default a [deployment group](/setup/deployments) updates every matching device at one pace. A **workflow** breaks that into ordered steps instead — a small canary batch, a sign-off, then everyone else — with each step choosing its own devices, its own concurrency, and how much failure it will tolerate before stopping. + +Workflows are defined in JSON and uploaded to a group, much as GitHub Actions and CircleCI pipelines are defined in YAML. + + + Workflows are an early release feature. Feedback goes to the [NervesHub issue tracker](https://github.com/nerves-hub/nerves_hub_web/issues). + + +## A worked example + +Four steps: canaries on fixed connections first, then cellular canaries, then one city, then a human decides whether the rest of the fleet follows. + +```json +{ + "version": 1, + "steps": [ + { + "name": "Canary", + "description": "wifi and ethernet canary devices", + "matching_conditions": { + "tags": ["canary"], + "network_interfaces": ["wifi", "ethernet"], + "match_limit": 20 + }, + "concurrent_updates": 10 + }, + { + "name": "Canary - LTE", + "description": "Cellular connected canaries", + "matching_conditions": { + "tags": ["canary"], + "network_interfaces": ["cellular"], + "match_limit": 10 + }, + "concurrent_updates": 10 + }, + { + "name": "Phoenix based", + "description": "Locally servicable devices", + "matching_conditions": { + "tags": ["city:phoenix"], + "match_limit": 100 + }, + "concurrent_updates": 25 + }, + { + "type": "approval_required", + "name": "Product sign-off", + "description": "Someone confirms the canaries are healthy" + } + ] +} +``` + +Splitting the canaries by connection is the point of that first pair: a firmware that breaks cellular is worth catching on ten metered devices rather than the whole fleet. + +## The file + +| Field | | +| --- | --- | +| `version` | Schema version. Required. | +| `steps` | Ordered list, **1 to 6** steps. Required. | + +Steps run in the order they are listed. Each takes: + +| Field | | +| --- | --- | +| `name` | Required on **every** step, including approval and catch-all steps. Max 50 characters. | +| `type` | `update_devices` (the default), `approval_required`, or `catch_all` | +| `description` | Optional, max 100 characters | +| `matching_conditions` | Which devices this step covers | +| `concurrent_updates` | How many of this step's devices update at once | +| `failure_tolerance` | How many may fail before the step fails | + +The smallest valid workflow is one named step: + +```json +{ "version": 1, "steps": [{ "name": "Everyone" }] } +``` + +## Step types + +| Type | What it does | +| --- | --- | +| `update_devices` | Updates the devices it matches, then moves on. The default when `type` is omitted. | +| `approval_required` | Halts the rollout until someone approves it | +| `catch_all` | Sweeps up every device not yet covered by an earlier step | + +## Choosing devices for a step + +`matching_conditions` narrows a step. Omitting a condition means it does not narrow anything. + +| Condition | | +| --- | --- | +| `tags` | A device must carry **every** tag listed | +| `network_interfaces` | The interface the device most recently connected over: `wifi`, `ethernet`, `cellular`, or `unknown` | +| `match_limit` | A hard cap on how many devices this step covers | + +`network_interfaces` is what makes it practical to hold metered devices back until a release has proven itself on cheap connections. + +## Pacing and failure + +`concurrent_updates` sets how many of a step's devices update at once — the step's own pace, independent of the others. + +`failure_tolerance` is how many of a step's devices may fail before the step itself fails and the rollout stops there rather than continuing into the next step. Give either a count or a percentage, not both. It defaults to one device, and a `catch_all` step never fails. + +Each step reports its own status as the rollout progresses: `waiting`, `in_progress`, `completed`, `skipped`, or `error`. + +## Uploading + +Open the deployment group, go to **Settings**, and use **Upload Workflow Definition** under Deployment Workflows. The file is validated on upload; if it is rejected nothing is stored and the error names the path that failed, so `steps/0` with `name` means the first step is missing its name. + +The two mistakes worth knowing about: + +- **Every step needs a `name`**, including `approval_required` and `catch_all` steps. It is easy to assume a step with no devices to match needs no name. +- **`version` and `steps` are both required**, and `steps` cannot be empty. + +Once stored, the group's Settings page reports how many steps the definition has. **Delete Workflow Definition** removes it and returns the group to updating every matching device at one pace. + +## Approving a step + +When a rollout reaches an `approval_required` step it stops and the deployment group shows a banner — *Waiting on you*, the step's name, and its description — with an **Approve and continue** button. Approving records who approved it and when, clears the banner, and the rollout proceeds to the next step. + +## What a workflow supersedes + +A workflow takes over two of the group's own [safety controls](/setup/deployments#rollout-safety-controls) while it is attached: + +| Group setting | While a workflow is attached | +| --- | --- | +| `concurrent_updates` | Not applied. Each step paces itself with its own `concurrent_updates`. | +| Priority queue | Not used. The step order decides which devices update first. | + +The failure and penalty box settings still apply — a workflow changes the order and pacing of a rollout, not what happens to a device that cannot take the update. diff --git a/concepts/deployments.mdx b/concepts/deployments.mdx deleted file mode 100644 index adf0b6d..0000000 --- a/concepts/deployments.mdx +++ /dev/null @@ -1,154 +0,0 @@ ---- -title: "How Deployment Groups Work" -sidebarTitle: "Deployment Groups" -description: "How deployment groups target devices by tag and version, how releases and workflow steps stage a rollout, and the controls that keep a bad update contained." ---- - -A deployment group is the primary mechanism for rolling out firmware to your fleet. It links a set of targeting rules to a sequence of firmware **releases**, giving you control over which devices update, when, and at what pace. Rather than pushing firmware to individual devices, you define the group's conditions once and NervesHub notifies every matching device as it comes online. - - - The web console calls these **deployment groups**. The REST API and the `nh` CLI still address them under `deployment` / `deployments` paths — for example `GET /api/orgs/{org}/products/{product}/deployments`. The two names refer to the same object. - - -## How Deployment Groups Work - -A deployment group holds targeting conditions and an ordered list of releases. Each release points at a firmware build (and optionally an archive), and the group tracks which release is current. - -When the group is active, NervesHub evaluates connected devices against its conditions. Matching devices receive a notification over their WebSocket channel telling them an update is available. The device then downloads the firmware and applies it — either immediately or at the next opportunity, depending on your `NervesHubLink.Client` configuration. - -Devices that are offline when a release goes out are evaluated the next time they connect, so a group keeps catching devices as they come back online with nothing to re-trigger manually. - - - A deployment group does not "finish". It stays active indefinitely, waiting for devices that match its conditions — including devices you register months later. Completion is tracked on the workflow steps within a release, not on the group itself. - - -## Targeting Devices - -A deployment group targets devices on two conditions, both of which must match: - -### Tags - -Tags are arbitrary strings you assign to devices — for example `main`, `qa`, `beta`, `factory`, or `region-us-east`. A group lists one or more tags and a **tag operator** that decides how they are matched: - -| Tag operator | Behaviour | -| ------------------------- | -------------------------------------------------------- | -| `Require all` *(default)* | A device matches only if it carries **every** listed tag | -| `Allow any` | A device matches if it carries **any** listed tag | - -So a group listing `beta` and `region-us-east` targets only devices carrying both under `Require all`, and every device carrying either one under `Allow any`. - -You assign tags when you register a device or update it later through the console, the API, or the CLI. Tags give you free-form grouping with no fixed hierarchy. - -### Version condition - -A group also filters on the firmware version a device is currently running. The version condition is a semver expression such as `>= 1.0.0`, `~> 1.2`, or `< 2.0.0`; only devices satisfying it are eligible. Leave it empty to match on tags alone. - -Version conditions let you orchestrate multi-step migration paths — requiring devices to be on `~> 1.x` before they can receive `2.0` — without devices on unsupported older versions picking up a breaking update. - - - A deployment group also records the **platform** and **architecture** of the firmware it was created with, and every later release must match them. This stops firmware for one board from being rolled out to a group full of another. - - -## Releases - -Firmware reaches a group through a release. Releases are numbered in the order they are created, each carries the firmware (and optional archive) being shipped plus an optional description and notes, and the group points at whichever one is current. - -Because the history is kept, you can see exactly which firmware a group shipped and when, and roll forward to a new release without losing that record. - -## Staged Rollouts with Workflows - -A release can carry a **workflow definition**: an ordered list of up to six steps that stages the rollout instead of updating every matching device at once. Steps run in order and come in three types: - -| Step type | What it does | -| ------------------- | ------------------------------------------------------------------------ | -| `update_devices` | Updates the devices this step matches, then moves on | -| `approval_required` | Halts the rollout until someone approves it | -| `catch_all` | Sweeps up every remaining device that has not yet been covered by a step | - -Each `update_devices` step narrows the devices it covers with its own matching conditions — any combination of: - -* **`tags`** — a device must carry every listed tag -* **`network_interfaces`** — the interface the device most recently connected over (`wifi`, `ethernet`, `cellular`, or `unknown`), so you can hold cellular devices back until a release is proven -* **`match_limit`** — a hard cap on how many devices the step covers - -A step also sets its own `concurrent_updates` and a `failure_tolerance`, expressed as either a number of devices or a percentage. Exceeding that tolerance fails the step and stops the rollout there rather than continuing into the next one. - -Each step reports its own status as the rollout progresses: `waiting`, `in_progress`, `completed`, `skipped`, or `error`. - -A typical shape is a small canary step, an approval gate, then a `catch_all`: - -1. `update_devices` — 20 devices tagged `canary`, 5 at a time -2. `approval_required` — an engineer confirms the canaries are healthy -3. `catch_all` — everything else - -## Rollout Safety Controls - -Beyond workflow steps, each group carries controls that limit the blast radius of a bad release: - -| Setting | Default | What it controls | -| ----------------------------- | ------- | ----------------------------------------------------------------------- | -| `concurrent_updates` | `10` | How many devices update at the same time | -| `device_failure_threshold` | `3` | Failures on a single device before it is put in the penalty box | -| `device_failure_rate_amount` | `5` | Failures on a single device within the rate window before penalising it | -| `device_failure_rate_seconds` | `180` | The rate window, in seconds | -| `failure_threshold` | `50` | Fleet-wide failures before the group is flagged unhealthy | -| `penalty_timeout_minutes` | `1440` | How long a penalised device waits before it may try again | -| `queue_management` | `FIFO` | Whether the update queue is drained oldest-first or newest-first | - -A device that trips its failure thresholds is placed in the **penalty box** and stops receiving update notifications until the timeout expires — which keeps one device stuck in a reboot loop from consuming rollout capacity. You can clear the penalty box manually from the device page. - -Refer to the [step-by-step setup guide](/setup/deployments) for where each of these lives in the UI and the values worth using in production. - -## Device Response to Updates - -When a device receives an update notification, the default behavior is to download and apply the firmware immediately. If you need custom logic — deferring updates during active use, checking battery level, or prompting a local UI — implement the `NervesHubLink.Client` behaviour in your application: - -```elixir -defmodule MyApp.NervesHubClient do - @behaviour NervesHubLink.Client - - @impl true - def update_available(data) do - if MyApp.safe_to_update?() do - :apply - else - # Check again in 60 seconds - {:reschedule, 60_000} - end - end -end -``` - -Configure your client module in `config/target.exs`: - -```elixir -config :nerves_hub_link, - client: MyApp.NervesHubClient -``` - -The `update_available/1` callback receives metadata about the available firmware (version, UUID, description) and must return `:apply` to proceed, `:ignore` to skip this notification, or `{:reschedule, ms}` to retry after a delay. - -## CLI Examples - -Create and manage deployment groups from the `nh` CLI: - -```bash -# Create a group targeting devices tagged "main" running firmware ~> 1.0 -nh deployment create --name "v2.0-rollout" --firmware --version "~> 1.0" --tag "main" - -# Activate the group so devices start receiving notifications -nh deployment update "v2.0-rollout" state on - -# Deactivate it to halt the rollout -nh deployment update "v2.0-rollout" state off -``` - -For a streamlined release workflow, use the `--deploy` flag on firmware upload to sign, upload, and attach the firmware in a single command: - -```bash -nh firmware upload my_project.fw --deploy "v2.0-rollout" -``` - - - The `--deploy` flag on `nh firmware upload` is the fastest path from a compiled `.fw` file to live devices. It uploads the firmware, registers it, and creates the release against the named group — no need to copy the firmware UUID and run a separate command. - diff --git a/docs.json b/docs.json index b8864e1..4d9c397 100644 --- a/docs.json +++ b/docs.json @@ -70,7 +70,7 @@ "pages": [ "concepts/pki", "concepts/firmware-signing", - "concepts/deployments" + "concepts/deployment-workflows" ] } ] @@ -133,6 +133,11 @@ "source": "/api/nerveshublink-client", "destination": "/integrations/nerves-hub-link", "permanent": true + }, + { + "source": "/concepts/deployments", + "destination": "/setup/deployments", + "permanent": true } ], "footer": { diff --git a/index.mdx b/index.mdx index f58b9d9..e0bd502 100644 --- a/index.mdx +++ b/index.mdx @@ -43,7 +43,7 @@ NervesHub gives you a complete platform for managing firmware updates and monito Monitor CPU, memory, load, and custom metrics across your entire fleet with configurable alarms. - + Roll a release out in stages — a canary batch, an approval gate, then everything else — with failure limits on each step. diff --git a/quickstart.mdx b/quickstart.mdx index 3572e2c..6f636cb 100644 --- a/quickstart.mdx +++ b/quickstart.mdx @@ -241,7 +241,7 @@ This guide uses `manage.nervescloud.com` as the host. Swap in your own if self-h 2. Set targeting **conditions** (version + tags). Leave tags empty for now so it matches your device. 3. Save and mark it **active**. - Full options: [How deployment groups work](/concepts/deployments). + Full options: [Deployment groups](/setup/deployments), including staged rollouts with [workflows](/concepts/deployment-workflows). With an active group pointing at newer firmware, NervesHub offers the update. The device downloads it, verifies the signature, applies it, and reboots into the new version. Track progress on the device's page. @@ -261,7 +261,7 @@ This guide uses `manage.nervescloud.com` as the host. Swap in your own if self-h Reduce update bandwidth dramatically by sending only the binary diff between firmware versions. - + Learn how to configure concurrency limits, failure thresholds, and staged rollouts. diff --git a/setup/deployments.mdx b/setup/deployments.mdx index 3a92ba3..c2a2af4 100644 --- a/setup/deployments.mdx +++ b/setup/deployments.mdx @@ -1,21 +1,15 @@ --- title: "Create and Activate Deployment Groups" sidebarTitle: "Deployment Groups" -description: "How to create, configure, and activate deployment groups that target specific devices by tag and version condition in NervesHub." +description: "Create deployment groups, target devices by tag and version, ship releases, and set the concurrency and failure limits that keep a bad update contained." --- -Deployment groups connect a firmware binary to the devices that should run it. A group specifies which firmware to deliver, which devices to target (by tag and current version), and how aggressively to roll it out. When a device checks in, NervesHub evaluates every active group against that device's tags and firmware version — if a match is found, the device receives an update notification and begins downloading. This page walks through creating and managing deployment groups from upload to activation. +A deployment group connects firmware to the devices that should run it. It holds the targeting rules — which devices are in scope — and a sequence of **releases**, each shipping one firmware build. When a device checks in, NervesHub evaluates every active group against that device's tags and firmware version, and a match earns an update notification. The web console calls these **deployment groups**, while the REST API and the `nh` CLI still use `deployment` / `deployments` in their paths and command names. They are the same object. - - To pause an in-progress rollout during an incident or if you observe unexpected device behavior, deactivate the group immediately with `nh deployment update "" state off`. Devices that have already applied the update are unaffected, but no further devices will receive it until you reactivate. - - -For a deeper explanation of how NervesHub evaluates targets, version conditions, releases, staged rollout workflows, concurrency limits, and failure thresholds, see the [Deployment groups concept guide](/concepts/deployments). - Before creating a group, you need a firmware UUID. If you have not yet uploaded your firmware, do so now: @@ -50,10 +44,6 @@ For a deeper explanation of how NervesHub evaluates targets, version conditions, | `--tag` | The device tag that qualifies a device for this group. Pass it more than once to list several tags. | Newly created groups are **inactive** by default. No devices will receive the update until you explicitly activate the group in the next step. - - - A group records the **platform** and **architecture** of the firmware it was created with. Every later release must match them, so create a separate group per board rather than reusing one. - @@ -74,25 +64,80 @@ For a deeper explanation of how NervesHub evaluates targets, version conditions, nh deployment show "v2.0-production" ``` - `nh deployment show` reports how many devices have been notified, how many have successfully applied the update, and how many are pending or have failed. Refresh the command periodically as the rollout progresses across your fleet. + `nh deployment show` reports how many devices have been notified, how many have successfully applied the update, and how many are pending or have failed. -## Shipping a New Release +## Targeting devices + +A group targets on two conditions, and a device must satisfy both. + +### Tags + +Tags are arbitrary strings you assign to devices — `main`, `qa`, `beta`, `region-us-east`. A group lists one or more, plus a **tag operator** deciding how they combine: + +| Tag operator | Behaviour | +| ------------------------- | -------------------------------------------------------- | +| `Require all` *(default)* | A device matches only if it carries **every** listed tag | +| `Allow any` | A device matches if it carries **any** listed tag | + +So a group listing `beta` and `region-us-east` targets only devices carrying both under `Require all`, and every device carrying either under `Allow any`. + +### Version condition + +The group also filters on the firmware version a device is currently running — a semver expression such as `>= 1.0.0`, `~> 1.2`, or `< 2.0.0`. Only devices satisfying it are eligible. Leave it empty to match on tags alone. + +Version conditions let you stage a migration: require devices to be on `~> 1.x` before they can receive `2.0`, so devices on unsupported older versions never pick up a breaking update. + + + A group also records the **platform** and **architecture** of the firmware it was created with, and every later release must match them. This stops firmware for one board being rolled out to a group full of another, so create a separate group per board rather than reusing one. + + +## Releases + +Firmware reaches a group through a release. Releases are numbered in the order they are created, each carries the firmware (and optional archive) being shipped plus an optional description and notes, and the group points at whichever is current. -To deliver a new firmware version through an existing group, update its firmware reference — this creates a new release on the group: +Because the history is kept, you can see exactly which firmware a group shipped and when. + + + A deployment group does not "finish". It stays active indefinitely, waiting for devices that match its conditions — including devices you register months later. + + +## Rollout safety controls + +Each group carries limits that contain a bad release: + +| Setting | Default | What it controls | +| ----------------------------- | ------- | ----------------------------------------------------------------------- | +| `concurrent_updates` | `10` | How many devices update at the same time | +| `device_failure_threshold` | `3` | Failures on a single device before it is put in the penalty box | +| `device_failure_rate_amount` | `5` | Failures on a single device within the rate window before penalising it | +| `device_failure_rate_seconds` | `180` | The rate window, in seconds | +| `failure_threshold` | `50` | Fleet-wide failures before the group is flagged unhealthy | +| `penalty_timeout_minutes` | `1440` | How long a penalised device waits before it may try again | +| `queue_management` | `FIFO` | Whether the update queue is drained oldest-first or newest-first | + +A device that trips its failure thresholds goes into the **penalty box** and stops receiving update notifications until the timeout expires, which keeps one device stuck in a reboot loop from consuming rollout capacity. Clear it manually from the device page once you have addressed the cause. + + + Attaching a [workflow](/concepts/deployment-workflows) supersedes two of these: each step paces itself with its own `concurrent_updates`, and the step order replaces the priority queue. + + +## Shipping a new release + +To deliver a new firmware version through an existing group, update its firmware reference — this creates a new release: ```bash nh deployment update "v2.0-production" firmware ``` -Devices are evaluated against the new release on their next check-in. You can also update the version condition or target tag in the same way: +Devices are evaluated against the new release on their next check-in. You can update the version condition or target tag the same way: ```bash nh deployment update "v2.0-production" tag "stable" ``` -## Deactivating a Group +## Deactivating a group Stop delivering updates to new devices without deleting the group: @@ -100,14 +145,26 @@ Stop delivering updates to new devices without deleting the group: nh deployment update "v2.0-production" state off ``` -Deactivating does not roll back devices that have already applied the firmware. Use this to pause a rollout, investigate an issue, or retire an old group gracefully. + + This is the fastest lever during an incident. Devices that have already applied the update are unaffected, but no further devices receive it until you reactivate. + -## One-Step Upload and Ship +Deactivating does not roll back devices that already applied the firmware. -For CI/CD pipelines where you always want the latest firmware to go out immediately, use the `--deploy` flag on the upload command: +## One-step upload and ship + +For CI/CD pipelines where the latest firmware should go out immediately, use `--deploy` on the upload command: ```bash nh firmware upload my_project.fw --deploy "v2.0-production" ``` -This uploads the firmware, creates a release on the named group, and activates it — all in a single command. The group must already exist; `--deploy` does not create one. +This uploads the firmware, creates a release on the named group, and activates it in one command. The group must already exist; `--deploy` does not create one. + +## Staging a rollout + +Everything above updates every matching device at the same pace. To roll out in stages instead — a small canary batch, a sign-off, then the rest — attach a workflow. + + + Define the stages of a rollout in a JSON file, with per-step targeting, concurrency, failure tolerance, and approval gates. +