Cut monitor-node CPU: raw health subs, idle-aware task tick - #96
Merged
Conversation
health_monitor and task_server were measured at 22% and 14-20% of a core on mote-01 during a nav mission, for logic that runs at 1-10 Hz. The premise was that they pay to deserialize high-rate messages. They do not, much: measured on the robot, deserialization is 13% of what a subscription costs and the rclpy wake-up is the other 87%. Both sanctioned Python fixes land, because both are free and both are real, paired against the unpatched build running at the same instant: health_monitor raw=True on the watched topics 18.1 -> 17.1 task_server idle tick 10 Hz -> 1 Hz 7.8 -> 6.9 The watches only count arrivals and never read a field, so they take bytes; the /diagnostics subscription stays deserialized because it reads status.name and status.level. A tree between missions ticks WaitForTask and nothing else, so it ticks at idle_tick_period until a command is accepted -- and _set_tick_rate resets the timer as well as re-periodding it, because setting a period does not move the expiry already pending, so without the reset the first tick of an accepted tree (the one that sends the Nav2 goal) waits out the rest of the idle period. Measured at 2.00 s, and held by a test. Two things had to be got right before any figure meant anything, and both are in tools/node_cpu.py, the sampler this leaves behind. The robot's own condition drifts in exactly the variable under study -- the drive servos answer intermittently, and a run where /tf was 33 Hz against one where it was 51 Hz reports the patch making everything worse, including slip_monitor, which was not touched. So a node is identified by the entry point its interpreter runs plus any __node:= rename, which lets two builds of one node be sampled against each other in a single run, and keeps the ros2 run and pixi run wrappers -- one of which does no work at all -- from being weighed instead of the node. The ~5% target is not reachable in Python and the C++ port of health_monitor is justified, on evidence the original framing did not have: a bare rclpy node is free (0.5%), four subscriptions carrying 101 msg/s cost 7.9 points more, and a TransformListener costs 4.8 for one 51 Hz stream -- the most expensive thing either node holds, and the bulk of what task_server spends while idle, through the listener AcquireObject creates in setup() and uses only during a fetch. health_monitor consumes ~152 msg/s, so ~12 points are gone before it does anything with them. Swapping the odometry TF watch for a topic watch buys nothing, both candidates being ~50 Hz. Rationale, rejected alternatives and what the port must be are in docs/tuning/2026-08-11-monitor-cpu.md, with the raw sampler output beside it. slip_monitor is now the largest remaining consumer at 7.1 and is filed separately: its maths is shared with tools/slip_replay.py, which is what set config/slip.yaml's thresholds, so a port forks that and invalidates the calibration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6o6x5Nn7QYUMxg2F3Wy2t
Comment on lines
+9
to
+10
| python -m mote_bringup.tools.node_cpu --duration 60 --tag idle \ | ||
| --out docs/tuning/2026-08-11-monitor-cpu/before |
There was a problem hiding this comment.
python -m mote_bringup.tools.node_cpu ... won't actually work: mote_bringup/tools/ has no __init__.py and isn't discovered by setup.py's find_packages(), so mote_bringup.tools isn't an importable module (running this raises No module named mote_bringup.tools). The same invalid form is repeated at line 34.
The working invocation — matching the node-cpu pixi task this PR adds (pixi.toml: node-cpu = "python mote_bringup/tools/node_cpu.py") — is:
python mote_bringup/tools/node_cpu.py --duration 60 --tag idle \
--out docs/tuning/2026-08-11-monitor-cpu/before
(and similarly for the --nodes example at line 34).
MJohnson459
added a commit
that referenced
this pull request
Aug 12, 2026
`python -m mote_bringup.tools.node_cpu` raises `ModuleNotFoundError: No module named 'mote_bringup.tools'`. `mote_bringup/tools/` has no `__init__.py`, so setup.py's `find_packages()` never picks it up -- deliberately, since these are harnesses run from a checkout and not code the robot installs. Every sibling tool documents the path form for that reason; node_cpu was the only one claiming otherwise, in its two usage examples and in the tuning run notes. Use `pixi run node-cpu`, which is the task this tool already ships with and the form camera_layer_decay documents (pixi appends trailing arguments, so `--summary`/`--nodes` pass straight through). Reported by review on #96. Claude-Session: https://claude.ai/code/session_01X6o6x5Nn7QYUMxg2F3Wy2t Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both sanctioned Python fixes landed, measured on mote-01, and the C++ decision is
taken on evidence — but the evidence is not what the task expected, and the ~5%
target is not reachable in Python. Branch
monitor-cpu-tuning, commit d588d08,unpushed. Write-up:
docs/tuning/2026-08-11-monitor-cpu.mdwith raw sampleroutput beside it.
What changed
health_monitor— the four watched topics are subscribedraw=True. Thewatches only count arrivals and never read a field.
/diagnosticsstaysdeserialized: it reads
status.name/status.level.task_server— newidle_tick_periodparameter (default 1.0). The tree ticksat
tick_periodonly while a task runs._set_tick_rateresets the timeras well as re-periodding it, because setting a period does not move the expiry
already pending — without the reset the first tick of an accepted tree (the one
that sends the Nav2 goal) waits out the rest of the idle period, measured at
2.00 s.
mote_bringup/tools/node_cpu.py(pixi run node-cpu) — the per-node samplerthe measurements were taken with, kept so they can be re-derived.
test_health_monitor_node.py(new — drives the real node and assertsthe watched topics are raw,
/diagnosticsis not, and a raw scan stillreaches
/healthand/diagnostics_agg), plus two cases intest_goto_tree.pyfor the tick rates.docs/tuningdoc, mkdocs nav entry,CLAUDE.md note.
Results (paired, both builds running at the same instant on mote-01)
health_monitortask_serverThe acceptance target of ~5% is not met, and cannot be met in Python. The
task's root-cause hypothesis was that these nodes pay to deserialize. Measured:
deserialization is 13% of what a subscription costs; the rclpy wake-up is the
other 87%, at ~0.78 ms of CPU per message delivered to a callback that increments
a counter. Decomposition (four probes, same live graph): bare rclpy node 0.5%;
+7.9 points for 4 subscriptions carrying 101 msg/s; +1.2 for deserializing them;
+4.8 for a single
TransformListeneron the 51 Hz/tf.health_monitorconsumes ~152 msg/s, so ~12 points are gone before it does anything.
Two corrections to the brief, both load-bearing:
it out ("
localizationis severity: info and low-rate, so it is not thecost"); a listener takes the whole
/tfstream regardless of which edges youask about. It is also most of what
task_serverspends while idle, through thelistener
AcquireObjectcreates insetup()and uses only during a fetch.task_server's cost. Dropping 10 Hz → 1 Hz removes 90%of the ticks and 12% of the node's CPU. The change is still right (free, and an
idling tree has nothing to advance) but it is not the lever the figure implied.
odometryTF watch for a topic watch) was measuredand rejected: both candidate topics are ~50 Hz, so it buys no wake-ups, and
a fresh topic is not evidence the TF edge Nav2 consumes was broadcast.
Verification
418 passed, 2 skippedacrossmote_bringup/test+mote_tasks/test; thefive
mote_fleete2e tests that constructTaskServerpass against a realmosquitto.
pre-commit run --all-filesclean.reset()removed it failswith "the accepted tree waits 2.00s for its first tick".
/healthand/diagnostics_aggunchanged in content and cadence (1.0 Hz,data: OK,moteroll-up first then one status per subsystem); the systemdwatchdog is still petted on every publish; command acceptance is unaffected
(it happens in the subscription callback, not on a tick).
Follow-ups filed
health_monitorto C++. Justified by the numbers, and speccedwith the correction above (it must own
/tftoo).slip_monitoris now the largest consumer at 7.1%. Not portable asit stands: its maths is shared with
slip_replay.py, which is what calibratedslip.yaml, so the shared-maths problem has to be answered first.Two things to know
MoteHardwarelogsFailed to read position from servo 7/9in bursts; the 50 Hz control loopcollapses to ~1.6 Hz,
/joint_statesto 1.6 Hz and/tfto 33 Hz. Itrecovered mid-session (3159 read failures in one run, 564 in the next). Arm
servos answered throughout, so it looks like the wheel servos specifically —
worth a look at the wiring. This is also why every result here is paired: a
sequential before/after over that drift reported the patch making everything
worse, including
slip_monitor, which was not touched.goto(it means the robot drives itself) and had no answer, so the figures arestack-up-idle with a synthetic 50 Hz
/joint_statesstanding in for a healthycontrol loop. The acceptance criterion is an idle one, and the paired design
makes the comparison sound regardless — but if you want the loaded numbers, say
so and it is one more run.
The Pi was left as found: stack down, scratch removed, its checkout untouched and
clean,
mote-agentstill active. Its code was never modified — the patchedmodules reached it through a scratch
PYTHONPATH.