docs: a design for seeing what the services say to each other - #52
Draft
pierre-rouanet wants to merge 2 commits into
Draft
docs: a design for seeing what the services say to each other#52pierre-rouanet wants to merge 2 commits into
pierre-rouanet wants to merge 2 commits into
Conversation
Two requests wear the name "monitor the message exchange": a developer at the bench asking where an operation stopped, and support asking whether a robot's IPC is working and who is using it. They are one event stream folded two ways, and building them separately is how a log and a counter end up disagreeing with no way to tell which lied. The tap goes at the dispatch boundary rather than on the socket. An interposing proxy is the cheap design and disqualifies itself twice: it reports its own uid to `may_mutate`, so it either breaks mutating calls or launders authorisation for every caller, and it can only ever see the unix socket — not the BLE leg, not mediad's gateway. Bytes on a wire also cannot say a call was refused by policy or a frame dropped because a subscriber lagged, which are the events worth having. Params stay opt-in and redacted through the typed Call, because the tap sees the line before the hand-written Debug impls that keep a customer's PSK and the pairing PIN out of the journal. The correlation id is the one wire change, and it goes first: it is the same category as min_supported and schema_version, cheap before clients exist and not retrofittable after. Assisted-by: Claude:claude-opus-5
Coverage73.88% lines on this branch. No base measurement to compare against. Per-file |
The shared serve loop moves from a deferred refactor to step one. The three loops differ in four ways that are requirements rather than drift — a peer policy in two of three, updaterd's 1 MiB line cap against 64 KiB, a push side in two of three, and two connection shapes — so this is a real abstraction over dispatcher, policy, cap and push stream, not a copy-paste removal. Doing it at three implementations rather than six is the argument for now. The extraction surfaces something that should not be answered in a serve-loop diff: robotd passes no policy, and is_mutating() does not cover robot.enable, so the call that starts a policy running on a walking robot is classified alongside hello. Recorded as an open question, with the note that classifying intents as mutating would deny padd unless its user is allowlisted. The correlation id now echoes on responses and notifications. The reason is that a captured line becomes self-describing: a log excerpt pasted into an issue can be reconstructed into a tree without also having the process memory that held the mapping. Cost is a field on every reply, omitted when absent, and about a kilobyte a second on a 50 Hz stream. Every error response reaches info, not only the mutating ones — which hands a retrying client the ability to evict a journal, so the loop collapses identical consecutive errors and demotes a connection past an error budget. The meter grows a crash record: the last snapshot plus a ring of the last N events, flushed to a fixed-size file under /var/lib. The ring is the part worth having — the trace cannot be left on permanently, and the events before a service died are exactly what nobody can go back and collect. Assisted-by: Claude:claude-opus-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design only — nothing implemented.
docs/design/monitor-design.md, plus a pointer fromarchitecture.md§8 and a row in the docs index.The shape
"Monitor the message exchange" is two requests: the trace (a developer at the bench, asking where an operation stopped) and the meter (support on a live robot, asking whether IPC is working and who is using it). One event stream, folded two ways — the same relationship
robotctl monitoralready has withrobot.state. §9.1 adds a third fold: the last of both, on disk, for whoever arrives after the service died.The decisions
The tap goes at the dispatch boundary, not on the socket. An interposing proxy is the cheap design and disqualifies itself twice:
may_mutatereadsSO_PEERCRED, which behind a proxy reports the proxy's uid for every caller — so it either breaks mutating calls or grants any peer permission to change the robot — and it can only see one of the four transports §4.1 commits us to. Bytes on a wire also cannot say a call was refused by policy, or that a frame was dropped because a subscriber lagged.The shared serve loop comes first, before any monitoring is written. A tap each new service must remember to call is exactly the
_ => Nonewildcardroute.rsrefuses on the grounds that it is safe in the moment and wrong over time. The three loops differ in four ways that are requirements rather than drift — a peer policy in two of three,updaterd's 1 MiB line cap against 64 KiB, a push side in two of three, two connection shapes — so this is a real abstraction over dispatcher, policy, cap and push stream. Underestimating it is the main risk in the plan, and three implementations is the cheapest it ever gets.Params are opt-in and redacted through the typed
Call. The tap sees the line before the hand-writtenDebugimpls that keep the wifi PSK and the pairing PIN out of the journal. At dispatch the redaction is the natural path; at the socket it is a thing to remember. The on-disk record never carries params at all.The correlation id echoes on responses and notifications, not only requests, so a captured line is self-describing — a log excerpt pasted into an issue reconstructs into a tree without the process memory that held the mapping. Omitted when absent; about a kilobyte a second on a 50 Hz stream. Needs a
PROTOCOL_VERSIONbump, so board and laptop move together.Every error response reaches
info, not only mutating ones, since read-only is most of the surface and its failures are invisible today. That hands a retrying client the ability to evict a journal, so the loop collapses identical consecutive errors and demotes a connection past an error budget — §8.1's retention argument applies here harder than it did there.The meter gets a crash record: last snapshot plus a ring of the last N events, flushed to a fixed-size file under
/var/lib, outside release dirs so it survives update and rollback. The ring is the part worth having — the trace cannot be left on permanently, and the events immediately before a service died are exactly what nobody can go back and collect.Surfaced by the refactor, deliberately not settled here
robotdpasses no peer policy, andCall::is_mutating()does not coverrobot.enable— so the call that starts a policy running on a walking robot is classified alongsidehello. Defensible, since safety is authoritative regardless of what a client asks, but it has never been written down as a decision and it should not be answered by accident in a diff about serve loops. Noted with the consequence if revisited:paddsendsrobot.enable, so classifying intents as mutating denies the gamepad unless its user is allowlisted.Other open questions: what
mediadchanges as the first service that is both client and gateway; how the trace ring is sized, givenpaddalone produces 50 events a second; and whether the shared loop is its own crate or aduck-ipc-protomodule, which is really a question about keeping the types free oftokio.