Updating a fleet of constrained nodes without bricking it.
Every conversation about edge AI eventually reaches the buyer's actual questions, and none of them are about the model. How do you update two hundred of these. What happens when an update is bad. What is inside the image. Can you rebuild the version that is running in the field today. The demonstrations are about inference; the procurement is about these four.
This repository is the unglamorous half, built as a working model rather than a slide: A/B slots with automatic rollback, digest-bound manifests, SBOM diffing, reproducible-build checking, and a waved rollout that halts itself.
pip install -e .
fol demo
plan : inspection-agent -> 2.4.0
waves : canary, pilot, fleet
updated : 2
failed : 4
rolled back : 2 <- the number that matters
untouched : 6
halted : in wave 'pilot' — 4/5 failed in wave 'pilot', budget was 20% (1 node(s))
✓ node-01 updated health probe passed
✓ node-02 updated health probe passed
✗ node-03 transport-failure node-03: link dropped during transfer
✗ node-04 digest-mismatch manifest says sha256:ceba2163…, bytes hash to sha256:c1735458…
✗ node-05 health-probe-failed health probe failed
✗ node-06 health-probe-failed no health probe supplied
Six nodes were never touched. That is the rollout working, not the rollout failing.
An update is provisional until the node says otherwise. Activation starts a
confirmation window. A node that boots a new image and does not check in — bad
image, dead network, someone pulled the power — reverts to the slot it was
running before, on its own timer, with nobody available to intervene. The state
that survives a power cut is PENDING, and PENDING reverts. Silence is a
rollback, never a success.
The fleet halts itself. A wave that exceeds its failure budget stops the rollout, and there is no flag to continue. The flag would be enabled at three in the morning by whoever has been awake for nineteen hours and wants the deployment finished, which is precisely when continuing is worst.
Everything else follows from those two.
| Concern | What is here | The failure it is aimed at |
|---|---|---|
| A/B slots | node.py — stage into the spare, activate provisionally, confirm or revert |
An update that overwrites the only working copy |
| Digest binding | artefact.py — the manifest carries the artefact's digest, checked on the node after transfer |
A transfer that succeeds and delivers the wrong bytes |
| Signature quorum | Manifest.verify_signature with a caller-supplied verifier |
An unsigned image, and — the subtler one — a check that was skipped because no verifier was configured |
| Health probes | update.py — pass confirms, fail rolls back, absent rolls back |
A node nobody can ask about keeping a change on the strength of silence |
| Waves and budgets | rollout.py — canary at zero budget, then widening waves |
A fleet-wide outage from one bad build |
| SBOM diff | sbom.py — added, removed, version-changed, and licence changes called out |
"What is in this image" asked on a deadline, sixty days too late |
| Reproducible builds | reproducible.py — hash both trees, name every file that differs |
An argument about what is actually deployed, at the start of every incident |
None of it knows what a node is. A node has slots, a digest, a health probe and a transport. Whether that is a Linux SBC with two rootfs partitions, an MCU with two flash banks, or a workload on a k3s cluster at the edge belongs to the deployment — see docs/ORCHESTRATORS.md for how the model maps onto RAUC, Mender, SWUpdate, balena and k3s.
Every stopping point has a stable code, because these are what end up grouped on a dashboard and quoted in an incident report:
transport-failure the bytes did not arrive; ordinary, and not an incident
digest-mismatch the bytes arrived and are not the bytes described
signature-invalid unsigned, under-signed, or no verifier was configured
slot-unavailable staging now would destroy the rollback target
not-staged activation attempted with nothing verified
health-probe-failed failed, raised, or absent
rollout-halted a wave exceeded its budget
fol demo --out run/rollout.json writes them as fleet-ops/rollout-report/v1.
Three things are yours to supply, and all three are deliberately not bundled:
transport = MyTransport() # put / get / has / delete
probe = lambda node: my_client.healthy(node.node_id)
verifier = lambda key_id, sig, payload: my_kms.verify(key_id, sig, payload)
report = Rollout(nodes, transport, probes, verifier, signature_quorum=2).run(
RolloutPlan.canary_then_rest("agent", "2.4.0", node_ids), artefact, manifest
)No cryptography ships here. Whoever runs a fleet already has a key story — a TPM, a cloud KMS, an HSM, an Ed25519 key in a file — and this package will not choose one on their behalf. What it insists on is that an absent verifier with a required quorum is a failure and never a skip.
| For | Read |
|---|---|
| Never used a terminal, want to see it work | GETTING_STARTED.md — assumes nothing, twenty minutes, no hardware |
| What it does and what counts as working | FUNCTIONAL_SPEC.md — actors, FR-1…FR-14, acceptance criteria |
| Writing code against it | TECHNICAL_REFERENCE.md — module by module, every artefact field |
| Why it is shaped this way | ARCHITECTURE.md — the argument, and what was rejected |
| A real board with two slots | BARE_METAL.md — U-Boot, MCUboot, watchdogs, the power-cut test |
| You already run RAUC, Mender or k3s | ORCHESTRATORS.md — the mapping, and what k3s does not give you |
| What is defended and what is not | THREAT_MODEL.md — adversaries A1–A4, residual risks R-1…R-6 |
| Every control and the test that proves it | CONTROL_MAP.md |
| How the suite is organised | TEST_STRATEGY.md — five tiers and the gate |
| What the first bench port will find wrong | PORTING.md |
| Decisions and their cost | docs/adr/ |
Five tiers, each answering a different question. The tier a test belongs to is the directory it lives in, and the marker is applied from the path — so a test cannot be moved between tiers and keep an old label.
make smoke # does it start at all 6 tests
make unit # each part at its boundary 57 tests
make functional # the specification, end to end 27 tests
make security # input and paths it did not create 22 tests
make pentest # attacks on the two rules 22 tests
make test # all of it 134 tests
make qa # ruff, mypy --strict, bandit, pip-audit, coverage >= 90%
The pen-test tier includes three tests that pass by demonstrating a gap: an older signed release replays successfully, a probe that passes then fails leaves a bad version committed, and a node that lies about its version is not detectable. Those are residual risks R-1 to R-3, and a pen-test suite in which the system always wins is a suite written after the fact.
- Not a device management platform. No enrolment, no inventory, no telemetry, no dashboard. Those are products and there are good ones.
- Not a container orchestrator. The waved rollout with a halt rule overlaps
with what a Kubernetes rollout does; where you have k3s, use it, and read
docs/ORCHESTRATORS.mdfor what it does not give you on a constrained node. - Not a signing service. See above.
- Not tested against hardware. Every node here is a Python object. The model
is the deliverable at this stage, and the first bench port will find something
wrong with it —
docs/PORTING.mdrecords what to expect.
Apache-2.0.