From d588d084fd1a606aedf628f969bd022226262dee Mon Sep 17 00:00:00 2001 From: Michael Johnson Date: Tue, 11 Aug 2026 21:45:38 +0100 Subject: [PATCH] Cut monitor-node CPU, and measure what the cost actually is health_monitor and task_server were measured at 22% and 14-20% of a core on mote-01 during a nav mission, for logic that runs at 1-10 Hz. The premise was that they pay to deserialize high-rate messages. They do not, much: measured on the robot, deserialization is 13% of what a subscription costs and the rclpy wake-up is the other 87%. Both sanctioned Python fixes land, because both are free and both are real, paired against the unpatched build running at the same instant: health_monitor raw=True on the watched topics 18.1 -> 17.1 task_server idle tick 10 Hz -> 1 Hz 7.8 -> 6.9 The watches only count arrivals and never read a field, so they take bytes; the /diagnostics subscription stays deserialized because it reads status.name and status.level. A tree between missions ticks WaitForTask and nothing else, so it ticks at idle_tick_period until a command is accepted -- and _set_tick_rate resets the timer as well as re-periodding it, because setting a period does not move the expiry already pending, so without the reset the first tick of an accepted tree (the one that sends the Nav2 goal) waits out the rest of the idle period. Measured at 2.00 s, and held by a test. Two things had to be got right before any figure meant anything, and both are in tools/node_cpu.py, the sampler this leaves behind. The robot's own condition drifts in exactly the variable under study -- the drive servos answer intermittently, and a run where /tf was 33 Hz against one where it was 51 Hz reports the patch making everything worse, including slip_monitor, which was not touched. So a node is identified by the entry point its interpreter runs plus any __node:= rename, which lets two builds of one node be sampled against each other in a single run, and keeps the ros2 run and pixi run wrappers -- one of which does no work at all -- from being weighed instead of the node. The ~5% target is not reachable in Python and the C++ port of health_monitor is justified, on evidence the original framing did not have: a bare rclpy node is free (0.5%), four subscriptions carrying 101 msg/s cost 7.9 points more, and a TransformListener costs 4.8 for one 51 Hz stream -- the most expensive thing either node holds, and the bulk of what task_server spends while idle, through the listener AcquireObject creates in setup() and uses only during a fetch. health_monitor consumes ~152 msg/s, so ~12 points are gone before it does anything with them. Swapping the odometry TF watch for a topic watch buys nothing, both candidates being ~50 Hz. Rationale, rejected alternatives and what the port must be are in docs/tuning/2026-08-11-monitor-cpu.md, with the raw sampler output beside it. slip_monitor is now the largest remaining consumer at 7.1 and is filed separately: its maths is shared with tools/slip_replay.py, which is what set config/slip.yaml's thresholds, so a port forks that and invalidates the calibration. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6o6x5Nn7QYUMxg2F3Wy2t --- CLAUDE.md | 2 + docs/tuning/2026-08-11-monitor-cpu.md | 180 ++++++++++++ .../ab-patched-vs-unpatched.csv | 121 ++++++++ .../ab-patched-vs-unpatched.json | 87 ++++++ .../2026-08-11-monitor-cpu/after-joints50.csv | 61 ++++ .../after-joints50.json | 63 +++++ .../2026-08-11-monitor-cpu/before-idle.csv | 61 ++++ .../2026-08-11-monitor-cpu/before-idle.json | 63 +++++ .../before-joints50.csv | 61 ++++ .../before-joints50.json | 63 +++++ .../2026-08-11-monitor-cpu/probe-tf.csv | 91 ++++++ .../2026-08-11-monitor-cpu/probe-tf.json | 87 ++++++ docs/tuning/2026-08-11-monitor-cpu/probe.csv | 121 ++++++++ docs/tuning/2026-08-11-monitor-cpu/probe.json | 75 +++++ docs/tuning/2026-08-11-monitor-cpu/runs.txt | 37 +++ mkdocs.yml | 1 + mote_bringup/mote_bringup/health_monitor.py | 15 +- mote_bringup/test/test_health_monitor_node.py | 138 +++++++++ mote_bringup/tools/node_cpu.py | 266 ++++++++++++++++++ mote_tasks/mote_tasks/task_server.py | 39 ++- mote_tasks/test/test_goto_tree.py | 81 ++++++ pixi.toml | 5 + 22 files changed, 1716 insertions(+), 2 deletions(-) create mode 100644 docs/tuning/2026-08-11-monitor-cpu.md create mode 100644 docs/tuning/2026-08-11-monitor-cpu/ab-patched-vs-unpatched.csv create mode 100644 docs/tuning/2026-08-11-monitor-cpu/ab-patched-vs-unpatched.json create mode 100644 docs/tuning/2026-08-11-monitor-cpu/after-joints50.csv create mode 100644 docs/tuning/2026-08-11-monitor-cpu/after-joints50.json create mode 100644 docs/tuning/2026-08-11-monitor-cpu/before-idle.csv create mode 100644 docs/tuning/2026-08-11-monitor-cpu/before-idle.json create mode 100644 docs/tuning/2026-08-11-monitor-cpu/before-joints50.csv create mode 100644 docs/tuning/2026-08-11-monitor-cpu/before-joints50.json create mode 100644 docs/tuning/2026-08-11-monitor-cpu/probe-tf.csv create mode 100644 docs/tuning/2026-08-11-monitor-cpu/probe-tf.json create mode 100644 docs/tuning/2026-08-11-monitor-cpu/probe.csv create mode 100644 docs/tuning/2026-08-11-monitor-cpu/probe.json create mode 100644 docs/tuning/2026-08-11-monitor-cpu/runs.txt create mode 100644 mote_bringup/test/test_health_monitor_node.py create mode 100755 mote_bringup/tools/node_cpu.py diff --git a/CLAUDE.md b/CLAUDE.md index 9ccbfe6..bc05fba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -246,6 +246,8 @@ Launch files, config, udev rules, NetworkManager drop-ins, systemd services, and **On-robot reliability** (see `mote_bringup/README.md`): `pixi run robot`/`mapping` include the health monitor, so a manual run publishes `/health` too; the systemd units are installed by `pixi run setup` but **not enabled** (autostart would drain the battery on a desk — opt in with `systemctl enable --now mote-bringup mote-health`). the systemd services restart with backoff and never permanently give up (`Restart=always`, `RestartSec`/`RestartSteps`/`RestartMaxDelaySec`, `StartLimitIntervalSec=0`), order after the udev-tagged `dev-mote_*.device` units, and bound the journal. A pre-flight self-check (`self_check.py`, run as `mote-bringup`'s `ExecStartPre`; `pixi run self-check`) gates bringup on servo ping + lidar/camera/disk/clock/config and keeps the robot idle with a clear reason on failure. A health monitor (`health_monitor.py`, `mote-health.service` with a `Type=notify` watchdog; `pixi run health`) publishes per-subsystem `diagnostic_msgs/DiagnosticArray` on `/diagnostics_agg` and a single OK/DEGRADED/FAULT summary on `/health`. **Wheel slip needs no IMU**: kinematic_icp takes wheel odometry as its prior and corrects it against the scan, so the correction already measures how wrong the wheels were — `slip_monitor.py` (started by `mote_launch.py` beside `system_monitor`) compares the two over a 1 s sliding window and publishes `slip` / `stuck` / `icp_fault` as the `slip` status on `/diagnostics`, plus the raw residual on `slip/residual`. All three are DEGRADED, never FAULT: each is a reason to stop and re-plan, not to refuse to drive, and a monitor that can halt the robot on a threshold is a worse failure than the slip. The maths is ROS-free in `odom_residual.py` and **shared with `tools/slip_replay.py`**, which is what set `config/slip.yaml`'s thresholds from the residual distribution over the real mapping bags — a threshold calibrated offline only means something if the robot computes the same number. Two things are load-bearing. **Only translation is thresholded**: the yaw residual is published but its p99 reaches the yaw rate itself, so no threshold survives a hard turn — and a lag sweep puts the two streams within ±10 ms, so this is scan-match jitter, *not* the stamp skew task 165 suspected. **A stalled lidar must not read as slip**: the window would freeze at the last pose while the wheels keep turning, growing without bound, so a source older than `max_lag` yields no verdict at all. `health_monitor` lifts named statuses off the shared `/diagnostics` by exact name, listed in `health.yaml`'s `diagnostic_statuses`. Six real events (two stuck robots, four scan-match excursions) were found in the existing "known-good" bags; the derivation and the sim demonstration are in `docs/tuning/2026-07-28-slip-detection.md`. Driver and nav2 nodes are `respawn=True` for per-node recovery under the whole-service systemd restart. Battery voltage is **not** software-measurable (the power bank exposes no telemetry); `system_monitor` reports the Pi's `get_throttled` flags as the only power signal — read via **`vcgencmd`**, since the Pi 4 sysfs node does not exist on a Pi 5, alongside the Active Cooler's `fan_rpm` from the `pwmfan` hwmon. +**What a Python monitor costs is its wake-ups, not its work** (`docs/tuning/2026-08-11-monitor-cpu.md`; `pixi run node-cpu` is the per-node sampler, and it can weigh two builds of one node against each other in a single run because hardware drifts between runs — this robot's servo bus answered in one run and not the next, moving `/tf` by 18 Hz and inverting a sequential before/after). Measured on mote-01: a bare rclpy node costs 0.5% of a core, four subscriptions carrying 101 msg/s cost 7.9 points more, and deserializing them costs only 1.2 on top — so **`raw=True` can only ever recover ~13%**, and `health_monitor`'s watches take it (they count arrivals and never read a field, the `/diagnostics` one excepted). A `TransformListener` is the most expensive thing a node can hold: it takes the whole 51 Hz `/tf` stream for 4.8 points whatever handful of edges it asks about, which is most of what `task_server` costs while idle (`AcquireObject` creates one in `setup()` and uses it only during a fetch). `task_server` therefore ticks its tree at `tick_period` only while a task runs and at `idle_tick_period` between them — worth 0.9 points, not the 90% the tick-rate drop suggests — and `_set_tick_rate` **resets** the timer as well as re-periodding it, because setting a period does not move the expiry already pending, so without the reset the first tick of an accepted tree waits out the rest of the idle period. The consequence for anything new here: a 1 Hz monitor written in Python cannot get under ~5% of a core while watching the robot's real topic rates — `health_monitor` sits at ~17 — and the fix is a language change, not a smarter callback. + **Launch hierarchy:** the two mission launches (`mapping_launch.py`, `robot_launch.py`) each take a `base` arg (default true) that includes the hardware base, and a `use_sim_time` arg they forward to everything they include. The sim runs these *same* files with `base:=false`, supplying a Gazebo base in place of the drivers — so the missions are defined once and the sim exercises the real launch files. - `robot_launch.py` — nav mission: `mote_launch.py` (if `base`) + `nav2_launch.py` (drive a saved map). Forwards a `map` arg, defaulting to the active site's map (see Sites). - `mapping_launch.py` — mapping mission: `mote_launch.py` (if `base`) + `slam_launch.py` + `nav2_launch.py` (`localisation:=false`) + `record_launch.py` (`streams:=mapping`, unless `record:=false`): build/extend a map with SLAM *and* drive to goals autonomously while doing so, recording the session for map provenance. diff --git a/docs/tuning/2026-08-11-monitor-cpu.md b/docs/tuning/2026-08-11-monitor-cpu.md new file mode 100644 index 0000000..2909b1f --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu.md @@ -0,0 +1,180 @@ +# Monitor-node CPU — the wake-up is the cost, not the payload, 2026-08-11 + +**Verdict: the two Python fixes landed and are worth ~1 point of a core each; +the ~5% target is not reachable in Python and the C++ port of `health_monitor` +is justified — on evidence that is not the evidence the task expected.** The +premise was that these nodes are expensive because they deserialize high-rate +messages. Measured on the robot, deserialization is **13%** of what a +subscription costs; the other 87% is the rclpy wake-up itself, which no change +to what the callback does can touch. + +Raw sampler output is in `2026-08-11-monitor-cpu/`; every figure below is a +`mean` from one of those JSON files and can be re-derived with +`mote_bringup/tools/node_cpu.py --summary `. + +## How it was measured + +`node_cpu.py` samples `/proc//stat` (utime+stime deltas) once a second per +node, and records the load average beside each sample. Run on **mote-01**, a +4-core Pi 5, with `pixi run robot` + `pixi run tasks` up and no task dispatched. + +Two things had to be got right before any number meant anything. + +**The robot's own condition drifts, and it drifts in exactly the variable under +study.** The drive servos answer intermittently: in the first run `MoteHardware` +logged 3159 `Failed to read position` and the 50 Hz control loop had collapsed to +1.6 Hz, taking `/joint_states` to 1.6 Hz and `/tf` to 33 Hz; in the next run the +same stack logged 564 and ran at 51 Hz. Since the monitors' cost is a function of +those rates, two sequential runs are not comparable — and the first sequential +before/after pair says the change made everything *worse*, including +`slip_monitor`, which was not changed at all: + +| | `health_monitor` | `task_server` | `slip_monitor` | load1 | +| --- | --- | --- | --- | --- | +| before (`/tf` 33 Hz) | 15.1 | 4.6 | 4.0 | 3.0 | +| after (`/tf` 51 Hz) | 17.3 | 7.1 | 7.3 | 3.2 | + +That table measures the servo bus, not the patch. Everything below is therefore +**paired**: both builds run at the same instant on the same machine, the second +one started renamed (`-r __node:=health_monitor_b`), so they see byte-identical +input. `node_cpu.py` identifies a node by the entry point its interpreter is +running plus any `__node:=` rename, which is what makes that possible — and what +keeps it from weighing the `ros2 run` and `pixi run` wrappers, whose command +lines repeat the node's own. + +**The 50 Hz `/joint_states` had to be restored.** With the servo bus failing, the +highest-rate topic `health_monitor` watches was not running at its real rate, so +a synthetic 50 Hz publisher stood in for a healthy control loop. It only restores +the arrival rate, which is all the monitor consumes. + +## What a subscription actually costs + +Four probe nodes, spun side by side against the same live graph +(`probe-tf.json`, 90 s, load1 3.6): + +| probe | what it holds | CPU % of a core | delta | +| --- | --- | --- | --- | +| `floor` | nothing — a bare rclpy node | 0.5 | — | +| `raw` | 4 subscriptions, `raw=True` (101 msg/s) | 8.4 | **+7.9** wake-ups | +| `deser` | the same 4, deserialized | 9.6 | **+1.2** deserialization | +| `tf` | one `TransformListener` (`/tf`, 51 msg/s) | 5.3 | **+4.8** | + +The four topics are `health_monitor`'s own: `/scan` and `/scan_filtered` at +10 Hz, `/joint_states` at 50 Hz, `/image_raw/compressed` at 29 Hz. + +Three things fall out of that table. + +**A bare rclpy node is free** (0.5%), so there is no fixed floor to blame and +nothing to reclaim by making a node do less between messages. + +**Deserialization is 1.2 of the 9.1 points a subscription set costs above the +floor — 13%.** The remaining 7.9 points is the executor round trip: wait set, +take, dispatch into Python. At 101 msg/s that is **0.78 ms of CPU to deliver one +message to a callback that increments a counter**, which on a ~2.4 GHz core is +around two million cycles. + +**A `TransformListener` is the single most expensive thing either node holds.** +It takes the whole `/tf` stream — 51 msg/s, whatever handful of edges the node +asks about — for 4.8 points, more than the camera and both lidar topics +together. + +Those numbers account for the nodes as measured. `health_monitor` at 16.9 is +0.5 floor + 7.9 topics + 4.8 TF + ~3.7 for the 1 Hz roll-up (two TF lookups, +eight `DiagnosticStatus` messages, two publishes). `task_server` at 6.9 is +0.5 floor + 4.8 TF — the listener `AcquireObject` creates in `setup()` and uses +only during a fetch — plus its idle subscriptions and its tick. + +## The two changes, measured + +`ab-patched-vs-unpatched.json`, 120 s, both builds running at once, load1 3.8: + +| node | before | after | saved | +| --- | --- | --- | --- | +| `health_monitor` (`raw=True` on the watched topics) | 18.1 | **17.1** | 1.0 | +| `task_server` (idle tick 10 Hz → 1 Hz) | 7.8 | **6.9** | 0.9 | + +Both are real and both are small, and the probe explains each independently. + +The 1.0 point on `health_monitor` is the deserialization the probe priced at +1.2 — the whole of what `raw=True` can ever be worth, since the wake-up it +cannot avoid is 87% of the cost. It is still worth keeping: it is free, and it +scales with payload size rather than count, so the ~29 fps `CompressedImage` is +most of it. + +The 0.9 points on `task_server` is the surprise. Dropping from 10 ticks a second +to 1 removes 90% of the ticks and 12% of the node's CPU, because **ticking was +never what `task_server` spent its time on**. 9 ticks/s of an idling tree cost +0.09 points each; the node's real load is the `/tf` stream it holds through +`AcquireObject`'s listener from the moment it starts until it exits. The idle +tick is still the right change — a tree between missions has nothing to advance, +and the saving is free — but it is not the lever the CPU figure suggested. + +Deadman behaviour is unchanged in both. `health_monitor` still pets the systemd +watchdog on every publish, and `/health` and `/diagnostics_agg` are byte-identical +in content and cadence (1.0 Hz, `data: OK`, `mote` roll-up first then one status +per subsystem — asserted in `test_health_monitor_node.py`). Command acceptance +never depended on the tick: `on_command` publishes `accepted:`/`rejected:` from +the subscription callback. What the idle rate could have delayed is the first +tick of the accepted tree, and does not, because `_set_tick_rate` resets the +timer as well as re-periodding it — **setting a period does not move the expiry +already pending**: a 5 s timer 0.3 s into its period still reports 4.7 s to go +after its period is set to 0.05 s, and 0.05 s after `reset()`. Without the reset +the first tick waits out the rest of the idle period (measured: 2.00 s), which +`test_goto_tree.py::test_idle_tick_rate_does_not_delay_the_mission` holds. + +## The C++ decision: taken, for `health_monitor` + +**The ~5% target is not reachable in Python.** `health_monitor` consumes ~152 +msg/s (101 topic + 51 TF). At the measured rclpy wake-up cost of ~0.078 points +per msg/s that is ~12 points before the node does anything with them, against a +5-point budget. Reaching 5% in Python would mean consuming **no more than ~55 +msg/s**, i.e. giving up watching the camera, the joint states, or odometry — +which is giving up the monitor. + +The alternatives were considered and rejected: + +* **Swap the `odometry` TF watch for a topic watch** (the task's stage-1 item 3). + It buys nothing: the candidate topics (`/diff_drive_controller/odom`, + `/tf`) are both ~50 Hz, so the wake-up count is unchanged. Dropping *both* TF + watches would retire the listener and its 4.8 points, but `localization` + (map→odom) has no topic equivalent, and a topic being fresh is not evidence + that the TF edge Nav2 consumes was broadcast. ~3 points for a weaker check is + a bad trade when C++ restores the real check cheaply. +* **Sampling the topics duty-cycled** — subscribe for 200 ms a second — cuts + wake-ups by 80% and makes a monitor that is not looking 80% of the time. +* **Liveliness QoS** would move freshness into the middleware and cost nothing + per message, but `MANUAL_BY_TOPIC` has to be asserted by the *publisher*, and + the publishers are sllidar, v4l2_camera and joint_state_broadcaster. + +What the port must be, on this evidence rather than on the original premise: + +* The lever is **the number of Python wake-ups**, so the port has to own *all* + the subscriptions including `/tf` — 4.8 of the 16.9 points are the TF + listener, which the original framing ruled out as "not the cost". +* `create_generic_subscription(topic, type_string, qos, cb)` takes the type name + as a runtime string exactly as `get_message(spec["type"])` does, so + `health.yaml`'s topic list survives the port unchanged, with no codegen. +* It needs a home. `mote_bringup` is `ament_python`; `mote_nav`'s charter is + "the C++ that runs inside other people's processes", which a standalone + `Type=notify` service node is not. A new small C++ package. +* Three Python pieces need counterparts, each a second implementation of + something that currently lives in one place: `mote_home.override()`, + `sd_notify.py`, and YAML loading (yaml-cpp). +* Equivalence bar: byte-identical `/diagnostics_agg` and `/health` content and + cadence on the same inputs, and `test_health_monitor.py`'s `_TopicWatch` / + `_TfWatch` cases ported to gtest — the roll-up rules are the behaviour, not + the subscriptions. + +That is a new first-party package plus three re-implementations, and it is +tracked separately rather than bolted onto this change. + +## `slip_monitor` is the largest remaining consumer + +At **7.1** (`ab-patched-vs-unpatched.json`) it now costs more than `task_server`. +It is out of scope here and is not a port candidate as it stands: its maths is +`mote_bringup/mote_bringup/odom_residual.py`, ROS-free and shared with `tools/slip_replay.py`, +and that sharing is what set `config/slip.yaml`'s thresholds from real bags. A +C++ port forks the maths and quietly invalidates the calibration, so the shared- +maths problem has to be answered first. Filed as a follow-up. + +For reference, `system_monitor` costs **0.6** and needs nothing. diff --git a/docs/tuning/2026-08-11-monitor-cpu/ab-patched-vs-unpatched.csv b/docs/tuning/2026-08-11-monitor-cpu/ab-patched-vs-unpatched.csv new file mode 100644 index 0000000..636200d --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/ab-patched-vs-unpatched.csv @@ -0,0 +1,121 @@ +t_s,health_monitor_cpu_pct,health_monitor_b_cpu_pct,task_server_cpu_pct,task_server_b_cpu_pct,slip_monitor_cpu_pct,system_monitor_cpu_pct,health_monitor_rss_mb,health_monitor_b_rss_mb,task_server_rss_mb,task_server_b_rss_mb,slip_monitor_rss_mb,system_monitor_rss_mb,load1 +1.0,14.0,16.0,5.0,7.0,6.0,0.0,76.2,75.7,88.1,87.2,74.4,68.5,5.41 +2.0,15.98,16.98,6.99,7.99,6.99,1.0,76.2,75.7,88.1,87.2,74.4,68.6,4.98 +3.0,17.98,17.98,7.99,8.99,7.99,0.0,76.2,75.7,88.1,87.2,74.4,68.6,4.98 +4.0,17.99,18.98,6.99,7.99,6.0,1.0,76.2,75.7,88.1,87.2,74.4,68.6,4.98 +5.0,17.99,16.99,6.0,7.99,6.99,0.0,76.2,75.7,88.1,87.2,74.4,68.6,4.98 +6.0,16.98,18.98,6.99,8.99,7.99,1.0,76.2,75.7,88.1,87.2,74.4,68.6,4.98 +7.0,17.98,17.98,6.99,6.99,6.99,1.0,76.2,75.7,88.1,87.2,74.4,68.6,5.06 +8.0,16.99,18.99,7.99,7.99,6.99,0.0,76.2,75.7,88.2,87.2,74.4,68.6,5.06 +9.0,17.99,17.99,6.99,7.99,7.99,1.0,76.2,75.7,88.2,87.2,74.4,68.6,5.06 +10.0,17.99,18.99,6.99,8.99,6.99,1.0,76.2,75.7,88.2,87.3,74.4,68.6,5.06 +11.0,17.99,18.98,6.99,6.99,6.99,1.0,76.2,75.8,88.2,87.3,74.4,68.6,5.06 +12.0,17.94,17.94,7.97,7.97,7.97,0.0,76.2,75.8,88.2,87.3,74.4,68.6,4.82 +13.0,17.98,17.98,6.99,7.99,5.99,1.0,76.2,75.8,88.2,87.3,74.4,68.6,4.82 +14.0,16.99,18.99,6.99,7.99,6.99,0.0,76.2,75.8,88.2,87.3,74.4,68.6,4.82 +15.0,16.98,17.98,6.99,7.99,7.99,1.0,76.2,75.8,88.2,87.3,74.4,68.6,4.82 +16.0,16.99,17.99,6.99,6.99,6.99,1.0,76.2,75.8,88.2,87.3,74.4,68.6,4.82 +17.0,15.99,18.99,6.99,7.99,6.99,0.0,76.2,75.8,88.2,87.3,74.5,68.6,4.59 +18.0,16.99,16.99,6.99,7.99,6.0,1.0,76.2,75.8,88.2,87.3,74.5,68.6,4.59 +19.0,16.99,17.98,6.99,7.99,6.99,0.0,76.2,75.8,88.2,87.3,74.5,68.6,4.59 +20.0,15.99,17.99,6.99,7.99,6.99,1.0,76.2,75.8,88.2,87.3,74.5,68.6,4.59 +21.0,17.98,16.99,6.99,7.99,6.99,1.0,76.2,75.8,88.2,87.3,74.5,68.6,4.59 +22.0,16.99,17.99,6.99,7.99,6.99,0.0,76.2,75.8,88.2,87.3,74.5,68.6,4.22 +23.0,16.99,17.98,6.99,8.99,6.99,1.0,76.2,75.8,88.2,87.3,74.5,68.6,4.22 +24.0,15.99,18.99,6.0,7.99,6.0,1.0,76.2,75.8,88.2,87.3,74.5,68.6,4.22 +25.0,17.98,17.98,8.99,7.99,7.99,0.0,76.2,75.8,88.2,87.3,74.5,68.6,4.22 +26.0,15.97,17.97,5.99,6.99,6.99,1.0,76.2,75.8,88.2,87.3,74.5,68.6,4.22 +27.0,17.98,17.98,7.99,7.99,6.99,1.0,76.2,75.8,88.2,87.3,74.5,68.6,3.89 +28.0,17.98,17.98,5.99,8.99,6.99,0.0,76.2,75.8,88.2,87.3,74.5,68.6,3.89 +29.0,15.98,18.98,7.99,6.99,7.99,1.0,76.2,75.8,88.2,87.3,74.5,68.6,3.89 +30.0,15.98,17.98,5.99,7.99,5.99,0.0,76.2,75.8,88.2,87.3,74.5,68.6,3.89 +31.0,17.98,17.98,7.99,7.99,6.99,1.0,76.2,75.8,88.2,87.3,74.5,68.6,3.89 +32.0,16.99,18.98,7.99,7.99,6.99,0.0,76.2,75.8,88.2,87.3,74.5,68.6,3.65 +33.0,18.98,18.98,6.99,7.99,6.99,2.0,76.2,75.8,88.2,87.3,74.5,68.6,3.65 +34.0,16.98,17.98,5.99,7.99,8.99,0.0,76.3,75.8,88.2,87.3,74.5,68.6,3.65 +35.0,17.98,18.98,7.99,7.99,6.99,1.0,76.3,75.8,88.2,87.3,74.5,68.7,3.65 +36.0,18.99,18.99,6.99,8.99,6.99,0.0,76.3,75.8,88.2,87.3,74.5,68.7,3.65 +37.0,17.98,17.98,6.99,7.99,7.99,1.0,76.3,75.8,88.2,87.3,74.5,68.7,3.44 +38.0,17.98,19.98,6.99,7.99,6.99,0.0,76.3,75.8,88.2,87.3,74.5,68.7,3.44 +39.0,18.98,17.98,6.99,7.99,5.99,1.0,76.3,75.8,88.2,87.3,74.5,68.7,3.44 +40.0,16.99,19.98,6.0,7.99,6.99,0.0,76.3,75.8,88.2,87.3,74.5,68.7,3.44 +41.0,17.99,17.99,7.99,7.99,7.99,1.0,76.3,75.8,88.2,87.3,74.5,68.7,3.44 +42.0,17.98,17.98,6.99,7.99,6.99,1.0,76.3,75.8,88.2,87.4,74.5,68.7,3.49 +43.0,16.98,18.98,7.99,6.99,6.99,1.0,76.3,75.8,88.3,87.4,74.5,68.7,3.49 +44.0,17.97,18.97,6.99,7.99,6.99,1.0,76.3,75.9,88.3,87.4,74.5,68.7,3.49 +45.0,16.99,17.99,6.0,7.99,6.99,0.0,76.3,75.9,88.3,87.4,74.5,68.7,3.49 +46.0,16.98,16.98,6.99,7.99,6.99,1.0,76.3,75.9,88.3,87.4,74.5,68.7,3.49 +47.0,16.98,18.98,6.99,7.99,6.99,0.0,76.3,75.9,88.3,87.4,74.5,68.7,3.85 +48.0,16.99,18.98,6.99,7.99,6.99,1.0,76.3,75.9,88.3,87.4,74.5,68.7,3.85 +49.0,16.98,16.98,6.99,7.99,7.99,0.0,76.3,75.9,88.3,87.4,74.5,68.7,3.85 +50.1,16.96,17.96,6.98,6.98,7.98,2.0,76.3,75.9,88.3,87.4,74.5,68.7,3.85 +51.1,17.98,19.98,6.99,8.99,7.99,0.0,76.3,75.9,88.3,87.4,74.6,68.7,3.85 +52.1,17.98,16.98,6.99,7.99,6.99,1.0,76.3,75.9,88.3,87.4,74.6,68.7,3.54 +53.1,16.98,18.98,6.99,7.99,6.99,0.0,76.3,75.9,88.3,87.4,74.6,68.7,3.54 +54.1,17.98,17.98,7.99,8.99,6.99,1.0,76.3,75.9,88.3,87.4,74.6,68.7,3.54 +55.1,16.99,17.99,6.99,6.99,6.99,0.0,76.3,75.9,88.3,87.4,74.6,68.7,3.54 +56.1,17.99,17.99,6.0,7.99,7.99,1.0,76.3,75.9,88.3,87.4,74.6,68.7,3.54 +57.1,16.99,18.99,7.99,7.99,6.99,1.0,76.3,75.9,88.3,87.4,74.6,68.7,3.26 +58.1,16.98,18.98,6.99,7.99,7.99,1.0,76.3,75.9,88.3,87.4,74.6,68.7,3.26 +59.1,18.98,17.99,6.99,8.99,7.99,0.0,76.3,75.9,88.3,87.4,74.6,68.7,3.26 +60.1,16.98,20.98,6.99,7.99,5.99,1.0,76.3,75.9,88.3,87.4,74.6,68.7,3.26 +61.1,17.98,18.98,7.99,7.99,8.99,0.0,76.3,75.9,88.3,87.4,74.6,68.7,3.26 +62.1,18.99,17.99,6.0,6.99,6.0,1.0,76.3,75.9,88.3,87.4,74.6,68.7,3.23 +63.1,15.99,18.99,6.99,8.99,6.99,1.0,76.3,75.9,88.3,87.4,74.6,68.7,3.23 +64.1,17.97,18.97,7.99,6.99,6.99,1.0,76.3,75.9,88.3,87.4,74.6,68.7,3.23 +65.1,16.98,17.98,5.99,6.99,6.99,0.0,76.3,75.9,88.3,87.4,74.6,68.7,3.23 +66.1,16.99,16.99,7.99,6.99,6.99,1.0,76.3,75.9,88.3,87.4,74.6,68.8,3.23 +67.1,16.98,18.97,6.99,7.99,6.99,0.0,76.3,75.9,88.3,87.4,74.6,68.8,3.14 +68.1,16.99,17.99,6.0,6.99,6.99,1.0,76.3,75.9,88.3,87.4,74.6,68.8,3.14 +69.1,16.99,17.99,6.99,6.99,7.99,1.0,76.3,75.9,88.3,87.4,74.6,68.8,3.14 +70.1,16.97,17.97,6.99,7.99,6.99,0.0,76.3,75.9,88.3,87.4,74.6,68.8,3.14 +71.1,16.99,18.99,7.99,6.99,7.99,1.0,76.3,75.9,88.3,87.4,74.6,68.8,3.14 +72.1,16.99,17.98,6.99,7.99,6.99,1.0,76.3,75.9,88.3,87.4,74.6,68.8,3.93 +73.1,16.99,18.99,6.99,6.99,6.99,0.0,76.3,75.9,88.3,87.4,74.6,68.8,3.93 +74.1,16.99,18.98,6.99,7.99,7.99,1.0,76.3,75.9,88.4,87.5,74.6,68.8,3.93 +75.1,16.99,17.99,6.99,7.99,6.99,0.0,76.3,76.0,88.4,87.5,74.6,68.8,3.93 +76.1,17.99,17.99,6.99,7.99,6.0,1.0,76.3,76.0,88.4,87.5,74.6,68.8,3.93 +77.1,16.96,17.96,6.98,6.98,6.98,1.0,76.3,76.0,88.4,87.5,74.6,68.8,3.93 +78.1,16.94,18.93,6.98,7.97,6.98,1.0,76.3,76.0,88.4,87.5,74.6,68.8,3.93 +79.1,16.97,18.97,7.99,8.99,6.99,1.0,76.3,76.0,88.4,87.5,74.6,68.8,3.93 +80.1,17.92,17.92,5.97,7.96,7.96,0.0,76.3,76.0,88.4,87.5,74.6,68.8,3.93 +81.1,16.99,17.99,6.99,7.99,6.99,1.0,76.3,76.0,88.4,87.5,74.6,68.8,3.93 +82.1,17.99,18.99,6.99,8.99,6.99,1.0,76.4,76.0,88.4,87.5,74.6,68.8,3.94 +83.1,16.99,17.99,6.0,7.99,7.99,0.0,76.4,76.0,88.4,87.5,74.6,68.8,3.94 +84.1,18.9,18.9,6.96,7.96,6.96,0.99,76.4,76.0,88.4,87.5,74.6,68.8,3.94 +85.1,18.98,18.98,6.99,7.99,6.99,0.0,76.4,76.0,88.4,87.5,74.6,68.8,3.94 +86.1,17.98,17.98,7.99,6.99,5.99,1.0,76.4,76.0,88.4,87.5,74.6,68.8,3.94 +87.1,16.99,17.99,6.99,8.99,7.99,0.0,76.4,76.0,88.4,87.5,74.6,68.8,3.7 +88.1,15.99,18.98,7.99,7.99,6.99,0.0,76.4,76.0,88.4,87.5,74.6,68.8,3.7 +89.1,15.91,14.91,4.97,5.97,6.96,1.99,76.4,76.0,88.4,87.5,74.6,68.8,3.7 +90.1,14.99,15.99,6.0,7.0,6.0,0.0,76.4,76.0,88.4,87.5,74.7,68.8,3.7 +91.1,13.97,14.97,6.99,6.99,5.99,1.0,76.4,76.0,88.4,87.5,74.7,68.8,3.7 +92.1,13.99,15.99,5.0,7.0,6.0,1.0,76.4,76.0,88.4,87.5,74.7,68.8,3.41 +93.1,15.99,14.99,6.0,7.0,6.0,0.0,76.4,76.0,88.4,87.5,74.7,68.8,3.41 +94.1,15.92,17.91,6.97,6.97,6.97,1.0,76.4,76.0,88.4,87.5,74.7,68.8,3.41 +95.1,17.98,17.98,6.99,8.99,6.99,0.0,76.4,76.0,88.4,87.5,74.7,68.8,3.41 +96.1,16.99,17.98,6.99,8.99,6.99,1.0,76.4,76.0,88.4,87.5,74.7,68.8,3.41 +97.1,15.98,17.98,6.99,6.99,6.99,0.0,76.4,76.0,88.4,87.5,74.7,68.8,3.29 +98.1,17.9,19.89,6.96,6.96,7.96,0.99,76.4,76.0,88.4,87.5,74.7,68.9,3.29 +99.1,17.98,17.98,7.99,8.99,6.99,0.0,76.4,76.0,88.4,87.5,74.7,68.9,3.29 +100.1,16.99,16.99,6.99,6.99,7.99,2.0,76.4,76.0,88.4,87.5,74.7,68.9,3.29 +101.1,15.99,18.98,6.99,7.99,6.99,0.0,76.4,76.0,88.4,87.5,74.7,68.9,3.29 +102.1,17.98,16.98,6.99,8.99,6.99,1.0,76.4,76.0,88.4,87.5,74.7,68.9,3.03 +103.1,15.99,17.98,6.99,7.99,7.99,0.0,76.4,76.0,88.4,87.6,74.7,68.9,3.03 +104.1,16.98,16.98,7.99,7.99,7.99,1.0,76.4,76.0,88.4,87.6,74.7,68.9,3.03 +105.1,17.98,17.98,5.99,7.99,6.99,0.0,76.4,76.0,88.4,87.6,74.7,68.9,3.03 +106.1,16.98,17.98,6.99,7.99,5.99,1.0,76.4,76.1,88.5,87.6,74.7,68.9,3.03 +107.1,14.99,15.99,5.99,6.99,6.99,1.0,76.4,76.1,88.5,87.6,74.7,68.9,3.27 +108.1,14.97,15.97,5.99,5.99,5.99,1.0,76.4,76.1,88.5,87.6,74.7,68.9,3.27 +109.1,13.99,16.99,5.0,7.99,6.0,0.0,76.4,76.1,88.5,87.6,74.7,68.9,3.27 +110.1,15.99,15.99,7.0,6.0,6.0,0.0,76.4,76.1,88.5,87.6,74.7,68.9,3.27 +111.1,15.94,16.94,5.98,5.98,5.98,1.0,76.4,76.1,88.5,87.6,74.7,68.9,3.27 +112.1,16.99,19.98,6.99,7.99,6.99,1.0,76.4,76.1,88.5,87.6,74.7,68.9,3.09 +113.1,17.99,17.99,6.99,7.99,6.99,0.0,76.4,76.1,88.5,87.6,74.7,68.9,3.09 +114.1,16.99,17.98,6.99,7.99,6.99,1.0,76.4,76.1,88.5,87.6,74.7,68.9,3.09 +115.1,16.99,18.99,6.99,7.99,6.99,1.0,76.4,76.1,88.5,87.6,74.7,68.9,3.09 +116.1,17.99,18.98,6.0,6.99,7.99,1.0,76.4,76.1,88.5,87.6,74.7,68.9,3.09 +117.1,16.98,18.98,7.99,7.99,6.99,0.0,76.4,76.1,88.5,87.6,74.7,68.9,2.84 +118.1,16.99,16.99,6.0,7.99,6.99,0.0,76.4,76.1,88.5,87.6,74.7,68.9,2.84 +119.1,17.98,18.98,6.99,6.99,6.99,1.0,76.4,76.1,88.5,87.6,74.7,68.9,2.84 +120.1,16.97,17.97,6.99,7.99,5.99,1.0,76.4,76.1,88.5,87.6,74.7,68.9,2.84 diff --git a/docs/tuning/2026-08-11-monitor-cpu/ab-patched-vs-unpatched.json b/docs/tuning/2026-08-11-monitor-cpu/ab-patched-vs-unpatched.json new file mode 100644 index 0000000..1b90b48 --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/ab-patched-vs-unpatched.json @@ -0,0 +1,87 @@ +{ + "csv": "/home/michael/tmp/monitor-cpu/ab/ab-patched-vs-unpatched.csv", + "samples": 120, + "span_s": 119.1, + "load1": { + "mean": 3.8, + "median": 3.6, + "p95": 5.1, + "max": 5.4 + }, + "cpu_pct": { + "health_monitor": { + "mean": 17.1, + "median": 17.0, + "p95": 19.0, + "max": 19.0 + }, + "health_monitor_b": { + "mean": 18.1, + "median": 18.0, + "p95": 19.9, + "max": 21.0 + }, + "task_server": { + "mean": 6.9, + "median": 7.0, + "p95": 8.0, + "max": 9.0 + }, + "task_server_b": { + "mean": 7.8, + "median": 8.0, + "p95": 9.0, + "max": 9.0 + }, + "slip_monitor": { + "mean": 7.1, + "median": 7.0, + "p95": 8.0, + "max": 9.0 + }, + "system_monitor": { + "mean": 0.6, + "median": 1.0, + "p95": 1.0, + "max": 2.0 + } + }, + "rss_mb": { + "health_monitor": { + "mean": 76.3, + "median": 76.3, + "p95": 76.4, + "max": 76.4 + }, + "health_monitor_b": { + "mean": 75.9, + "median": 75.9, + "p95": 76.1, + "max": 76.1 + }, + "task_server": { + "mean": 88.3, + "median": 88.3, + "p95": 88.5, + "max": 88.5 + }, + "task_server_b": { + "mean": 87.4, + "median": 87.4, + "p95": 87.6, + "max": 87.6 + }, + "slip_monitor": { + "mean": 74.6, + "median": 74.6, + "p95": 74.7, + "max": 74.7 + }, + "system_monitor": { + "mean": 68.7, + "median": 68.7, + "p95": 68.9, + "max": 68.9 + } + } +} diff --git a/docs/tuning/2026-08-11-monitor-cpu/after-joints50.csv b/docs/tuning/2026-08-11-monitor-cpu/after-joints50.csv new file mode 100644 index 0000000..80bb5e5 --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/after-joints50.csv @@ -0,0 +1,61 @@ +t_s,health_monitor_cpu_pct,task_server_cpu_pct,slip_monitor_cpu_pct,system_monitor_cpu_pct,health_monitor_rss_mb,task_server_rss_mb,slip_monitor_rss_mb,system_monitor_rss_mb,load1 +1.0,14.99,6.0,7.0,1.0,74.8,86.8,73.1,68.1,2.77 +2.0,13.97,5.99,5.99,0.0,74.8,86.8,73.1,68.1,2.77 +3.0,14.98,5.99,5.99,1.0,74.8,86.8,73.1,68.1,2.77 +4.0,13.98,5.99,5.99,0.0,74.8,86.8,73.1,68.1,2.77 +5.0,13.99,6.0,6.0,0.0,74.8,86.8,73.1,68.1,2.77 +6.0,18.99,7.0,7.99,1.0,74.8,86.8,73.1,68.1,2.78 +7.0,16.99,7.99,7.0,0.0,74.8,86.8,73.2,68.1,2.78 +8.0,16.99,7.99,7.0,1.0,74.8,86.8,73.2,68.1,2.78 +9.0,17.98,6.99,7.99,1.0,74.9,86.8,73.2,68.1,2.78 +10.0,16.99,6.0,6.99,0.0,74.9,86.8,73.2,68.1,2.78 +11.0,17.99,7.0,7.0,1.0,74.9,86.8,73.2,68.1,2.56 +12.0,17.97,7.99,7.99,1.0,74.9,86.8,73.2,68.1,2.56 +13.0,16.99,7.0,7.0,1.0,74.9,86.8,73.2,68.1,2.56 +14.0,17.99,7.0,7.0,0.0,74.9,86.8,73.2,68.2,2.56 +15.0,16.99,7.99,7.99,1.0,74.9,86.8,73.2,68.2,2.56 +16.0,16.99,7.0,7.0,1.0,74.9,86.8,73.2,68.2,4.12 +17.0,17.99,7.0,7.99,0.0,74.9,86.8,73.2,68.2,4.12 +18.0,16.99,7.0,7.99,1.0,74.9,86.8,73.2,68.2,4.12 +19.0,17.99,6.99,6.99,0.0,74.9,86.8,73.2,68.2,4.12 +20.0,16.99,7.99,6.99,2.0,74.9,86.8,73.2,68.2,4.12 +21.0,17.99,7.0,6.0,0.0,74.9,86.8,73.2,68.2,3.95 +22.0,17.99,7.0,7.99,0.0,74.9,86.9,73.2,68.2,3.95 +23.0,17.99,6.0,7.0,1.0,74.9,86.9,73.2,68.2,3.95 +24.0,17.98,6.99,7.99,1.0,74.9,86.9,73.2,68.2,3.95 +25.0,17.96,7.98,6.98,0.0,74.9,86.9,73.2,68.2,3.95 +26.0,16.96,7.98,7.98,1.0,74.9,86.9,73.2,68.2,3.63 +27.0,17.99,6.0,7.99,1.0,74.9,86.9,73.2,68.2,3.63 +28.0,15.99,7.99,7.0,1.0,74.9,86.9,73.2,68.2,3.63 +29.0,17.99,6.99,6.99,0.0,74.9,86.9,73.2,68.2,3.63 +30.0,17.99,7.0,7.0,1.0,74.9,86.9,73.2,68.2,3.63 +31.0,17.93,6.97,7.97,0.0,74.9,86.9,73.2,68.2,3.42 +32.0,16.99,7.99,7.99,2.0,74.9,86.9,73.2,68.2,3.42 +33.0,18.99,6.0,7.0,0.0,74.9,86.9,73.2,68.2,3.42 +34.0,17.99,6.99,6.99,0.0,74.9,86.9,73.2,68.2,3.42 +35.0,16.99,7.0,7.0,0.0,74.9,86.9,73.2,68.2,3.42 +36.0,16.99,7.0,7.99,2.0,74.9,86.9,73.2,68.2,3.15 +37.0,16.99,7.99,6.0,0.0,74.9,86.9,73.2,68.2,3.15 +38.0,17.98,6.99,7.99,1.0,74.9,86.9,73.2,68.2,3.15 +39.0,17.99,7.0,7.0,0.0,74.9,86.9,73.2,68.2,3.15 +40.0,16.99,7.99,7.0,1.0,74.9,86.9,73.3,68.2,3.15 +41.0,17.96,6.98,7.98,0.0,74.9,86.9,73.3,68.2,3.21 +42.0,16.92,6.97,6.97,1.0,74.9,86.9,73.3,68.2,3.21 +43.0,15.99,6.99,6.99,1.0,74.9,86.9,73.3,68.2,3.21 +44.0,17.99,7.0,6.0,1.0,74.9,86.9,73.3,68.2,3.21 +45.0,16.99,7.0,7.99,0.0,74.9,86.9,73.3,68.2,3.21 +46.0,16.99,7.0,7.99,1.0,74.9,86.9,73.3,68.2,3.04 +47.1,18.98,7.99,6.99,0.0,75.0,86.9,73.3,68.2,3.04 +48.1,16.95,7.98,7.98,1.99,75.0,86.9,73.3,68.3,3.04 +49.1,16.98,5.99,6.99,0.0,75.0,86.9,73.3,68.3,3.04 +50.1,16.99,7.99,7.99,0.0,75.0,86.9,73.3,68.3,3.04 +51.1,17.99,6.99,7.99,2.0,75.0,86.9,73.3,68.3,2.79 +52.1,16.99,7.0,7.0,0.0,75.0,86.9,73.3,68.3,2.79 +53.1,17.99,7.0,7.99,0.0,75.0,86.9,73.3,68.3,2.79 +54.1,16.99,6.99,6.0,1.0,75.0,86.9,73.3,68.3,2.79 +55.1,17.99,7.0,7.99,0.0,75.0,86.9,73.3,68.3,2.79 +56.1,17.99,7.99,7.0,2.0,75.0,86.9,73.3,68.3,2.65 +57.1,15.99,7.0,7.99,0.0,75.0,86.9,73.3,68.3,2.65 +58.1,16.99,7.0,7.99,1.0,75.0,86.9,73.3,68.3,2.65 +59.1,17.99,7.0,7.0,1.0,75.0,86.9,73.3,68.3,2.65 +60.1,17.99,7.0,7.99,1.0,75.0,86.9,73.3,68.3,2.65 diff --git a/docs/tuning/2026-08-11-monitor-cpu/after-joints50.json b/docs/tuning/2026-08-11-monitor-cpu/after-joints50.json new file mode 100644 index 0000000..e2371f3 --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/after-joints50.json @@ -0,0 +1,63 @@ +{ + "csv": "/home/michael/tmp/monitor-cpu/after/after-joints50.csv", + "samples": 60, + "span_s": 59.1, + "load1": { + "mean": 3.2, + "median": 3.1, + "p95": 4.1, + "max": 4.1 + }, + "cpu_pct": { + "health_monitor": { + "mean": 17.3, + "median": 17.0, + "p95": 19.0, + "max": 19.0 + }, + "task_server": { + "mean": 7.1, + "median": 7.0, + "p95": 8.0, + "max": 8.0 + }, + "slip_monitor": { + "mean": 7.3, + "median": 7.0, + "p95": 8.0, + "max": 8.0 + }, + "system_monitor": { + "mean": 0.6, + "median": 1.0, + "p95": 2.0, + "max": 2.0 + } + }, + "rss_mb": { + "health_monitor": { + "mean": 74.9, + "median": 74.9, + "p95": 75.0, + "max": 75.0 + }, + "task_server": { + "mean": 86.9, + "median": 86.9, + "p95": 86.9, + "max": 86.9 + }, + "slip_monitor": { + "mean": 73.2, + "median": 73.2, + "p95": 73.3, + "max": 73.3 + }, + "system_monitor": { + "mean": 68.2, + "median": 68.2, + "p95": 68.3, + "max": 68.3 + } + } +} diff --git a/docs/tuning/2026-08-11-monitor-cpu/before-idle.csv b/docs/tuning/2026-08-11-monitor-cpu/before-idle.csv new file mode 100644 index 0000000..f1276b7 --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/before-idle.csv @@ -0,0 +1,61 @@ +t_s,health_monitor_cpu_pct,task_server_cpu_pct,slip_monitor_cpu_pct,system_monitor_cpu_pct,health_monitor_rss_mb,task_server_rss_mb,slip_monitor_rss_mb,system_monitor_rss_mb,load1 +1.0,10.0,4.0,4.0,0.0,74.6,86.3,72.7,68.1,2.32 +2.0,10.99,6.0,5.0,0.0,74.6,86.3,72.7,68.1,2.32 +3.0,9.99,5.0,5.0,1.0,74.6,86.3,72.7,68.1,2.32 +4.0,9.0,4.0,4.0,0.0,74.6,86.3,72.7,68.1,2.32 +5.0,8.99,5.0,4.0,1.0,74.6,86.3,72.7,68.1,2.32 +6.0,10.99,4.0,4.0,1.0,74.6,86.3,72.7,68.1,2.13 +7.0,9.99,5.0,4.0,0.0,74.6,86.3,72.7,68.1,2.13 +8.0,13.99,5.0,5.0,0.0,74.6,86.3,72.7,68.1,2.13 +9.0,11.99,5.0,4.0,0.0,74.6,86.3,72.7,68.1,2.13 +10.0,11.99,4.0,5.0,2.0,74.6,86.3,72.7,68.1,2.13 +11.0,10.99,6.0,5.0,0.0,74.6,86.4,72.7,68.1,1.96 +12.0,12.99,4.0,6.0,1.0,74.6,86.4,72.7,68.1,1.96 +13.0,11.99,5.0,5.0,0.0,74.6,86.4,72.7,68.2,1.96 +14.0,11.99,5.0,4.0,1.0,74.6,86.4,72.7,68.2,1.96 +15.0,12.99,4.0,5.0,0.0,74.6,86.4,72.7,68.2,1.96 +16.0,12.99,5.99,4.99,0.0,74.6,86.4,72.7,68.2,1.96 +17.0,11.99,5.0,5.0,1.0,74.6,86.4,72.7,68.2,1.96 +18.0,11.99,5.0,5.0,1.0,74.6,86.4,72.7,68.2,1.96 +19.0,11.99,5.0,4.0,1.0,74.6,86.4,72.8,68.2,1.96 +20.0,11.98,5.99,4.99,0.0,74.6,86.4,72.8,68.2,1.96 +21.0,11.99,6.0,6.0,1.0,74.6,86.4,72.8,68.2,1.81 +22.0,11.99,6.0,4.0,0.0,74.6,86.4,72.8,68.2,1.81 +23.0,11.99,5.0,5.0,0.0,74.6,86.4,72.8,68.2,1.81 +24.0,12.99,6.0,5.0,1.0,74.7,86.4,72.8,68.2,1.81 +25.0,10.99,5.0,4.0,1.0,74.7,86.4,72.8,68.2,1.81 +26.0,10.99,4.99,4.99,1.0,74.7,86.4,72.8,68.2,3.1 +27.0,11.99,6.0,4.0,0.0,74.7,86.4,72.8,68.2,3.1 +28.0,11.99,6.0,5.0,0.0,74.7,86.4,72.8,68.2,3.1 +29.0,11.99,5.0,5.0,2.0,74.7,86.4,72.8,68.2,3.1 +30.0,10.99,5.0,5.0,0.0,74.7,86.4,72.8,68.2,3.1 +31.0,11.99,5.99,4.0,0.0,74.7,86.4,72.8,68.2,2.93 +32.0,10.99,5.0,4.0,1.0,74.7,86.4,72.8,68.2,2.93 +33.0,11.99,6.0,5.0,0.0,74.7,86.4,72.8,68.2,2.93 +34.0,10.99,5.0,4.0,0.0,74.7,86.4,72.8,68.2,2.93 +35.0,9.99,4.0,4.0,1.0,74.7,86.4,72.8,68.2,2.93 +36.0,9.99,4.0,3.0,1.0,74.7,86.4,72.8,68.2,2.7 +37.0,9.99,5.0,5.0,0.0,74.7,86.4,72.8,68.2,2.7 +38.0,6.99,3.99,2.0,0.0,74.7,86.4,72.8,68.2,2.7 +39.0,10.99,5.0,5.0,1.0,74.7,86.4,72.8,68.2,2.7 +40.0,11.96,4.98,4.98,1.0,74.7,86.4,72.8,68.2,2.7 +41.0,11.97,5.98,4.99,0.0,74.7,86.4,72.8,68.2,2.64 +42.0,11.98,5.99,4.99,0.0,74.7,86.4,72.8,68.2,2.64 +43.0,11.99,5.0,5.0,1.0,74.7,86.4,72.8,68.2,2.64 +44.0,11.99,6.0,5.0,1.0,74.7,86.4,72.8,68.2,2.64 +45.0,10.95,5.97,3.98,0.0,74.7,86.4,72.8,68.2,2.64 +46.0,12.99,5.0,5.0,1.0,74.7,86.4,72.8,68.2,2.59 +47.0,9.99,5.0,3.0,0.0,74.7,86.4,72.8,68.2,2.59 +48.0,8.99,4.0,4.0,0.0,74.7,86.4,72.8,68.2,2.59 +49.0,9.99,4.0,5.0,1.0,74.7,86.4,72.8,68.2,2.59 +50.0,9.99,5.0,4.0,1.0,74.7,86.4,72.8,68.2,2.59 +51.0,10.99,3.0,4.0,1.0,74.7,86.5,72.8,68.3,2.54 +52.0,11.99,6.99,5.0,0.0,74.7,86.5,72.8,68.3,2.54 +53.0,10.99,5.0,4.0,1.0,74.7,86.5,72.8,68.3,2.54 +54.0,11.99,5.0,5.0,0.0,74.7,86.5,72.8,68.3,2.54 +55.0,11.99,5.0,5.0,0.0,74.7,86.5,72.8,68.3,2.54 +56.0,11.99,5.99,4.0,0.0,74.7,86.5,72.8,68.3,2.34 +57.1,9.99,6.0,5.0,1.0,74.7,86.5,72.8,68.3,2.34 +58.1,9.99,5.0,4.0,0.0,74.7,86.5,72.8,68.3,2.34 +59.1,9.99,5.0,4.0,2.0,74.8,86.5,72.9,68.3,2.34 +60.1,10.99,5.0,4.0,0.0,74.8,86.5,72.9,68.3,2.34 diff --git a/docs/tuning/2026-08-11-monitor-cpu/before-idle.json b/docs/tuning/2026-08-11-monitor-cpu/before-idle.json new file mode 100644 index 0000000..72cd922 --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/before-idle.json @@ -0,0 +1,63 @@ +{ + "csv": "/home/michael/tmp/monitor-cpu/before/before-idle.csv", + "samples": 60, + "span_s": 59.1, + "load1": { + "mean": 2.4, + "median": 2.4, + "p95": 3.1, + "max": 3.1 + }, + "cpu_pct": { + "health_monitor": { + "mean": 11.3, + "median": 12.0, + "p95": 13.0, + "max": 14.0 + }, + "task_server": { + "mean": 5.1, + "median": 5.0, + "p95": 6.0, + "max": 7.0 + }, + "slip_monitor": { + "mean": 4.5, + "median": 5.0, + "p95": 5.0, + "max": 6.0 + }, + "system_monitor": { + "mean": 0.5, + "median": 0.0, + "p95": 2.0, + "max": 2.0 + } + }, + "rss_mb": { + "health_monitor": { + "mean": 74.7, + "median": 74.7, + "p95": 74.7, + "max": 74.8 + }, + "task_server": { + "mean": 86.4, + "median": 86.4, + "p95": 86.5, + "max": 86.5 + }, + "slip_monitor": { + "mean": 72.8, + "median": 72.8, + "p95": 72.8, + "max": 72.9 + }, + "system_monitor": { + "mean": 68.2, + "median": 68.2, + "p95": 68.3, + "max": 68.3 + } + } +} diff --git a/docs/tuning/2026-08-11-monitor-cpu/before-joints50.csv b/docs/tuning/2026-08-11-monitor-cpu/before-joints50.csv new file mode 100644 index 0000000..7f63fb0 --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/before-joints50.csv @@ -0,0 +1,61 @@ +t_s,health_monitor_cpu_pct,task_server_cpu_pct,slip_monitor_cpu_pct,system_monitor_cpu_pct,health_monitor_rss_mb,task_server_rss_mb,slip_monitor_rss_mb,system_monitor_rss_mb,load1 +1.0,12.0,4.0,3.0,0.0,76.7,88.4,75.6,70.0,3.06 +2.0,12.99,4.0,3.0,0.0,76.7,88.4,75.6,70.0,3.06 +3.0,11.99,3.0,4.0,2.0,76.7,88.4,75.6,70.0,3.06 +4.0,10.99,4.0,3.0,0.0,76.7,88.4,75.6,70.0,2.81 +5.0,12.99,4.0,3.0,1.0,76.7,88.4,75.6,70.0,2.81 +6.0,14.98,4.99,4.99,0.0,76.7,88.4,75.6,70.0,2.81 +7.0,15.99,4.0,4.0,0.0,76.7,88.4,75.6,70.0,2.81 +8.0,15.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,2.81 +9.0,14.99,5.0,5.0,1.0,76.7,88.4,75.6,70.0,2.67 +10.0,15.99,4.0,4.0,0.0,76.7,88.4,75.6,70.0,2.67 +11.0,14.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,2.67 +12.0,15.98,4.99,4.0,1.0,76.7,88.4,75.6,70.0,2.67 +13.0,15.99,4.0,4.0,0.0,76.7,88.4,75.6,70.0,2.67 +14.0,15.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,2.45 +15.0,15.98,4.0,4.99,1.0,76.7,88.4,75.6,70.0,2.45 +16.0,14.99,4.0,4.0,0.0,76.7,88.4,75.6,70.0,2.45 +17.0,15.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,2.45 +18.0,14.99,5.0,4.0,0.0,76.7,88.4,75.6,70.0,2.45 +19.0,14.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,3.22 +20.0,15.99,4.0,5.0,1.0,76.7,88.4,75.6,70.0,3.22 +21.0,14.99,5.0,4.0,0.0,76.7,88.4,75.6,70.0,3.22 +22.0,14.99,5.0,3.0,0.0,76.7,88.4,75.6,70.0,3.22 +23.0,15.99,5.0,5.0,0.0,76.7,88.4,75.6,70.0,3.22 +24.0,14.99,5.0,4.0,2.0,76.7,88.4,75.6,70.0,2.96 +25.0,14.99,4.0,3.0,0.0,76.7,88.4,75.6,70.0,2.96 +26.0,15.99,5.0,5.0,1.0,76.7,88.4,75.6,70.0,2.96 +27.0,13.99,5.0,3.0,0.0,76.7,88.4,75.6,70.0,2.96 +28.0,15.99,4.0,4.0,1.0,76.7,88.4,75.6,70.0,2.96 +29.0,14.99,4.0,4.0,0.0,76.7,88.4,75.6,70.0,2.72 +30.0,15.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,2.72 +31.0,16.98,5.0,4.0,1.0,76.7,88.4,75.6,70.0,2.72 +32.0,13.98,3.99,3.99,0.0,76.7,88.4,75.6,70.0,2.72 +33.0,14.99,5.0,5.0,1.0,76.7,88.4,75.6,70.0,2.72 +34.0,14.98,4.99,3.99,1.0,76.7,88.4,75.6,70.0,2.5 +35.0,15.99,4.0,4.0,0.0,76.7,88.4,75.6,70.0,2.5 +36.0,14.99,4.0,3.0,1.0,76.7,88.4,75.6,70.0,2.5 +37.0,15.99,5.0,5.0,0.0,76.7,88.4,75.6,70.0,2.5 +38.0,14.98,3.99,3.99,1.0,76.7,88.4,75.6,70.0,2.5 +39.0,14.99,5.0,4.0,0.0,76.7,88.4,75.6,70.0,2.3 +40.0,14.99,4.0,4.0,1.0,76.7,88.4,75.6,70.0,2.3 +41.0,14.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,2.3 +42.0,14.99,6.0,4.0,0.0,76.7,88.4,75.6,70.0,2.3 +43.0,14.98,4.0,4.0,1.0,76.7,88.4,75.6,70.0,2.3 +44.0,14.99,5.0,4.0,0.0,76.7,88.4,75.6,70.0,4.2 +45.0,14.99,5.0,4.0,0.0,76.7,88.4,75.6,70.0,4.2 +46.0,14.99,4.0,4.0,1.0,76.7,88.4,75.6,70.0,4.2 +47.0,14.99,5.0,5.0,1.0,76.7,88.4,75.6,70.0,4.2 +48.0,15.99,4.0,4.0,0.0,76.7,88.4,75.6,70.0,4.2 +49.0,14.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,3.86 +50.0,14.99,4.0,4.0,1.0,76.7,88.4,75.6,70.0,3.86 +51.0,15.99,5.0,4.0,0.0,76.7,88.4,75.6,70.0,3.86 +52.0,15.99,4.0,4.0,1.0,76.7,88.4,75.6,70.0,3.86 +53.0,14.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,3.86 +54.0,14.99,5.0,4.0,0.0,76.7,88.4,75.6,70.0,3.56 +55.0,15.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,3.56 +56.0,15.98,4.0,4.0,0.0,76.7,88.4,75.6,70.0,3.56 +57.0,14.99,5.0,4.0,1.0,76.7,88.4,75.6,70.0,3.56 +58.0,14.99,5.0,4.0,0.0,76.7,88.4,75.6,70.0,3.56 +59.0,15.99,4.0,4.0,0.0,76.7,88.4,75.6,70.0,3.27 +60.0,14.99,5.0,4.0,2.0,76.7,88.4,75.6,70.0,3.27 diff --git a/docs/tuning/2026-08-11-monitor-cpu/before-joints50.json b/docs/tuning/2026-08-11-monitor-cpu/before-joints50.json new file mode 100644 index 0000000..e3b7fe7 --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/before-joints50.json @@ -0,0 +1,63 @@ +{ + "csv": "/home/michael/tmp/monitor-cpu/before/before-joints50.csv", + "samples": 60, + "span_s": 59.0, + "load1": { + "mean": 3.0, + "median": 2.9, + "p95": 4.2, + "max": 4.2 + }, + "cpu_pct": { + "health_monitor": { + "mean": 15.1, + "median": 15.0, + "p95": 16.0, + "max": 17.0 + }, + "task_server": { + "mean": 4.6, + "median": 5.0, + "p95": 5.0, + "max": 6.0 + }, + "slip_monitor": { + "mean": 4.0, + "median": 4.0, + "p95": 5.0, + "max": 5.0 + }, + "system_monitor": { + "mean": 0.6, + "median": 1.0, + "p95": 2.0, + "max": 2.0 + } + }, + "rss_mb": { + "health_monitor": { + "mean": 76.7, + "median": 76.7, + "p95": 76.7, + "max": 76.7 + }, + "task_server": { + "mean": 88.4, + "median": 88.4, + "p95": 88.4, + "max": 88.4 + }, + "slip_monitor": { + "mean": 75.6, + "median": 75.6, + "p95": 75.6, + "max": 75.6 + }, + "system_monitor": { + "mean": 70.0, + "median": 70.0, + "p95": 70.0, + "max": 70.0 + } + } +} diff --git a/docs/tuning/2026-08-11-monitor-cpu/probe-tf.csv b/docs/tuning/2026-08-11-monitor-cpu/probe-tf.csv new file mode 100644 index 0000000..67c836a --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/probe-tf.csv @@ -0,0 +1,91 @@ +t_s,cpu_probe_floor_cpu_pct,cpu_probe_raw_cpu_pct,cpu_probe_deser_cpu_pct,cpu_probe_tf_cpu_pct,health_monitor_cpu_pct,task_server_cpu_pct,cpu_probe_floor_rss_mb,cpu_probe_raw_rss_mb,cpu_probe_deser_rss_mb,cpu_probe_tf_rss_mb,health_monitor_rss_mb,task_server_rss_mb,load1 +1.0,0.0,6.99,8.99,5.0,15.99,6.0,64.9,69.3,70.4,69.1,79.0,89.3,3.18 +2.0,1.0,8.97,9.97,4.98,17.94,6.98,64.9,69.3,70.4,69.1,79.0,89.3,3.18 +3.0,0.0,7.99,9.99,5.99,15.99,7.99,64.9,69.3,70.4,69.1,79.0,89.3,3.18 +4.0,0.0,8.99,8.99,5.0,17.98,5.99,64.9,69.3,70.4,69.1,79.0,89.3,2.93 +5.0,2.0,7.99,9.99,5.0,16.98,7.99,64.9,69.3,70.4,69.1,79.0,89.3,2.93 +6.0,0.0,9.99,9.99,5.0,15.98,6.99,64.9,69.3,70.4,69.1,79.0,89.3,2.93 +7.0,1.0,7.97,9.96,5.98,18.93,6.98,64.9,69.3,70.4,69.1,79.0,89.3,2.93 +8.0,0.0,7.99,7.99,4.99,14.98,6.99,64.9,69.3,70.4,69.1,79.0,89.3,2.93 +9.0,1.0,6.99,9.99,5.0,14.99,5.0,64.9,69.3,70.4,69.1,79.0,89.3,2.69 +10.0,0.0,7.0,7.99,5.0,14.99,7.0,64.9,69.3,70.4,69.1,79.0,89.3,2.69 +11.0,0.0,7.0,7.99,4.0,15.99,6.0,64.9,69.3,70.4,69.1,79.0,89.3,2.69 +12.0,0.0,7.99,8.99,4.0,11.99,6.0,64.9,69.3,70.4,69.1,79.0,89.3,2.69 +13.0,1.0,7.99,8.99,6.0,17.99,6.99,64.9,69.3,70.4,69.1,79.0,89.3,2.69 +14.0,0.0,8.99,8.99,5.0,16.99,7.99,64.9,69.3,70.4,69.1,79.0,89.3,4.8 +15.0,1.0,7.99,10.99,5.0,16.99,6.99,65.0,69.3,70.4,69.1,79.0,89.3,4.8 +16.0,0.0,8.99,9.99,6.0,17.99,6.99,65.0,69.4,70.4,69.1,79.0,89.3,4.8 +17.0,1.0,7.99,8.99,4.99,14.98,6.99,65.0,69.4,70.4,69.1,79.0,89.3,4.8 +18.0,0.0,7.99,7.99,4.0,15.99,5.0,65.0,69.4,70.4,69.1,79.0,89.3,4.8 +19.0,0.0,7.0,7.99,5.0,14.99,7.0,65.0,69.4,70.4,69.1,79.0,89.3,4.41 +20.0,2.0,7.99,9.99,5.0,14.99,5.0,65.0,69.4,70.4,69.1,79.0,89.3,4.41 +21.0,0.0,7.99,6.99,4.0,13.99,6.0,65.0,69.4,70.4,69.1,79.0,89.3,4.41 +22.0,0.0,7.99,9.99,5.0,15.99,6.0,65.0,69.4,70.5,69.1,79.0,89.3,4.41 +23.0,1.0,8.99,9.99,5.0,17.99,7.99,65.0,69.4,70.5,69.2,79.0,89.3,4.41 +24.0,0.0,7.99,9.99,6.99,17.98,6.99,65.0,69.4,70.5,69.2,79.0,89.3,4.14 +25.0,0.0,7.99,8.99,5.0,16.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,4.14 +26.0,2.0,8.99,9.99,5.0,16.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,4.14 +27.0,0.0,8.97,10.97,4.98,16.95,6.98,65.0,69.4,70.5,69.2,79.0,89.3,4.14 +28.0,0.0,7.99,8.99,5.99,16.98,5.99,65.0,69.4,70.5,69.2,79.0,89.3,4.14 +29.0,1.0,9.96,9.96,4.98,16.93,6.97,65.0,69.4,70.5,69.2,79.0,89.3,3.81 +30.0,0.0,7.99,9.98,4.99,17.97,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.81 +31.0,0.0,8.99,9.99,5.99,16.98,7.99,65.0,69.4,70.5,69.2,79.0,89.3,3.81 +32.0,1.0,7.99,9.99,6.0,16.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.81 +33.0,0.0,8.99,8.99,5.0,17.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.81 +34.0,1.0,8.99,9.99,5.0,16.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.74 +35.0,0.0,6.99,8.99,6.0,17.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.74 +36.0,0.0,8.99,10.99,5.0,17.99,7.99,65.0,69.4,70.5,69.2,79.0,89.3,3.74 +37.0,1.0,8.99,8.99,6.0,16.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.74 +38.0,0.0,7.99,9.99,5.0,18.98,5.99,65.0,69.4,70.5,69.2,79.0,89.3,3.74 +39.0,1.0,8.99,8.99,5.0,15.99,7.99,65.0,69.4,70.5,69.2,79.0,89.3,3.52 +40.0,1.0,8.99,9.99,5.99,18.97,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.52 +41.0,0.0,8.99,9.99,5.0,16.99,7.99,65.0,69.4,70.5,69.2,79.0,89.3,3.52 +42.0,1.0,8.99,8.99,6.0,16.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.52 +43.0,1.0,7.99,10.99,5.0,18.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.52 +44.1,0.0,8.97,8.97,4.99,18.94,6.98,65.0,69.4,70.5,69.2,79.0,89.3,3.4 +45.1,0.0,8.99,9.99,6.0,16.99,6.99,65.0,69.4,70.5,69.2,79.0,89.3,3.4 +46.1,1.0,8.99,10.99,5.0,18.98,7.99,65.1,69.4,70.5,69.2,79.0,89.3,3.4 +47.1,0.0,7.99,9.99,4.99,15.98,5.99,65.1,69.4,70.5,69.2,79.0,89.3,3.4 +48.1,1.0,8.99,8.99,6.0,18.99,6.99,65.1,69.4,70.5,69.3,79.0,89.3,3.4 +49.1,0.0,8.99,9.99,5.0,17.98,6.99,65.1,69.4,70.5,69.3,79.0,89.3,3.21 +50.1,1.0,8.98,9.98,5.99,16.96,7.98,65.1,69.4,70.5,69.3,79.0,89.3,3.21 +51.1,0.0,8.98,9.98,4.99,16.97,5.99,65.1,69.4,70.5,69.3,79.0,89.3,3.21 +52.1,0.0,7.99,8.99,5.0,16.99,7.99,65.1,69.4,70.5,69.3,79.0,89.3,3.21 +53.1,0.0,8.99,9.99,6.0,16.99,6.0,65.1,69.4,70.5,69.3,79.0,89.3,3.21 +54.1,1.0,8.99,10.99,4.99,16.98,6.99,65.1,69.5,70.5,69.3,79.0,89.3,2.95 +55.1,0.0,7.99,9.99,5.0,16.99,6.99,65.1,69.5,70.5,69.3,79.0,89.3,2.95 +56.1,2.0,8.99,9.99,5.99,16.98,7.99,65.1,69.5,70.5,69.3,79.0,89.3,2.95 +57.1,0.0,8.98,8.98,5.99,15.97,6.99,65.1,69.5,70.5,69.3,79.0,89.3,2.95 +58.1,0.0,7.99,8.99,6.0,16.99,6.99,65.1,69.5,70.5,69.3,79.0,89.3,2.95 +59.1,1.0,7.99,9.99,4.99,16.98,6.99,65.1,69.5,70.5,69.3,79.0,89.3,2.72 +60.1,0.0,8.99,9.99,4.99,17.98,6.99,65.1,69.5,70.5,69.3,79.0,89.3,2.72 +61.1,0.0,7.99,9.99,5.0,17.98,6.99,65.1,69.5,70.5,69.3,79.0,89.3,2.72 +62.1,1.0,8.97,8.97,5.98,16.95,6.98,65.1,69.5,70.6,69.3,79.0,89.3,2.72 +63.1,0.0,8.99,9.99,5.99,17.98,7.99,65.1,69.5,70.6,69.3,79.0,89.3,2.72 +64.1,1.0,7.99,9.99,5.0,16.98,6.99,65.1,69.5,70.6,69.3,79.0,89.3,2.82 +65.1,0.0,8.99,9.99,5.0,17.98,6.99,65.1,69.5,70.6,69.3,79.0,89.3,2.82 +66.1,0.0,8.99,9.99,4.99,16.98,7.99,65.1,69.5,70.6,69.3,79.0,89.3,2.82 +67.1,1.0,8.97,8.97,6.98,16.95,6.98,65.1,69.5,70.6,69.3,79.0,89.3,2.82 +68.1,1.0,8.99,10.99,4.0,17.98,6.99,65.1,69.5,70.6,69.3,79.0,89.3,2.82 +69.1,0.0,8.99,8.99,5.99,15.99,6.99,65.1,69.5,70.6,69.3,79.0,89.3,2.83 +70.1,1.0,8.99,9.99,5.99,18.97,6.99,65.1,69.5,70.6,69.3,79.0,89.3,2.83 +71.1,0.0,8.99,9.99,4.99,16.98,6.99,65.1,69.5,70.6,69.3,79.0,89.3,2.83 +72.1,0.0,8.99,9.99,5.0,16.99,6.99,65.1,69.5,70.6,69.3,79.0,89.3,2.83 +73.1,2.0,8.98,9.98,4.99,16.96,7.98,65.1,69.5,70.6,69.3,79.0,89.3,2.83 +74.1,0.0,7.99,9.99,4.99,17.98,7.99,65.1,69.5,70.6,69.3,79.0,89.3,4.37 +75.1,0.0,8.99,9.99,5.99,17.97,6.99,65.1,69.5,70.6,69.3,79.0,89.3,4.37 +76.1,1.0,7.99,9.99,5.99,16.98,6.99,65.1,69.5,70.6,69.3,79.0,89.3,4.37 +77.1,0.0,7.99,8.99,4.0,14.99,5.99,65.1,69.5,70.6,69.3,79.0,89.3,4.37 +78.1,0.0,7.99,8.99,5.0,14.99,6.0,65.2,69.5,70.6,69.4,79.0,89.3,4.37 +79.1,1.0,7.98,7.98,4.99,14.97,5.99,65.2,69.5,70.6,69.4,79.0,89.3,4.34 +80.1,1.0,6.99,7.99,5.0,14.99,5.99,65.2,69.5,70.6,69.4,79.0,89.3,4.34 +81.1,0.0,7.0,7.99,4.0,13.99,7.0,65.2,69.5,70.6,69.4,79.0,89.3,4.34 +82.1,0.0,7.99,9.99,5.0,15.99,6.0,65.2,69.5,70.6,69.4,79.0,89.3,4.34 +83.1,1.0,7.99,8.99,6.0,16.99,7.99,65.2,69.5,70.6,69.4,79.0,89.3,4.34 +84.1,0.0,7.99,9.99,5.0,17.99,6.0,65.2,69.5,70.6,69.4,79.0,89.3,3.99 +85.1,1.0,8.99,9.99,4.99,18.98,7.99,65.2,69.6,70.6,69.4,79.0,89.3,3.99 +86.1,0.0,8.99,9.99,5.99,17.98,7.99,65.2,69.6,70.6,69.4,79.0,89.3,3.99 +87.1,1.0,8.99,9.99,5.0,17.99,6.99,65.2,69.6,70.6,69.4,79.0,89.3,3.99 +88.1,0.0,7.99,9.99,6.0,16.99,6.99,65.2,69.6,70.7,69.4,79.0,89.3,3.99 +89.1,0.0,8.99,9.99,5.0,16.99,6.99,65.2,69.6,70.7,69.4,79.0,89.3,4.23 +90.1,1.0,8.99,10.99,5.0,16.99,5.99,65.2,69.6,70.7,69.4,79.0,89.3,4.23 diff --git a/docs/tuning/2026-08-11-monitor-cpu/probe-tf.json b/docs/tuning/2026-08-11-monitor-cpu/probe-tf.json new file mode 100644 index 0000000..739e92b --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/probe-tf.json @@ -0,0 +1,87 @@ +{ + "csv": ".claude/worktrees/monitor-cpu/docs/tuning/2026-08-11-monitor-cpu/probe-tf.csv", + "samples": 90, + "span_s": 89.1, + "load1": { + "mean": 3.6, + "median": 3.5, + "p95": 4.8, + "max": 4.8 + }, + "cpu_pct": { + "cpu_probe_floor": { + "mean": 0.5, + "median": 0.0, + "p95": 2.0, + "max": 2.0 + }, + "cpu_probe_raw": { + "mean": 8.4, + "median": 9.0, + "p95": 9.0, + "max": 10.0 + }, + "cpu_probe_deser": { + "mean": 9.6, + "median": 10.0, + "p95": 11.0, + "max": 11.0 + }, + "cpu_probe_tf": { + "mean": 5.3, + "median": 5.0, + "p95": 6.0, + "max": 7.0 + }, + "health_monitor": { + "mean": 16.9, + "median": 17.0, + "p95": 19.0, + "max": 19.0 + }, + "task_server": { + "mean": 6.9, + "median": 7.0, + "p95": 8.0, + "max": 8.0 + } + }, + "rss_mb": { + "cpu_probe_floor": { + "mean": 65.0, + "median": 65.0, + "p95": 65.2, + "max": 65.2 + }, + "cpu_probe_raw": { + "mean": 69.4, + "median": 69.4, + "p95": 69.6, + "max": 69.6 + }, + "cpu_probe_deser": { + "mean": 70.5, + "median": 70.5, + "p95": 70.6, + "max": 70.7 + }, + "cpu_probe_tf": { + "mean": 69.2, + "median": 69.2, + "p95": 69.4, + "max": 69.4 + }, + "health_monitor": { + "mean": 79.0, + "median": 79.0, + "p95": 79.0, + "max": 79.0 + }, + "task_server": { + "mean": 89.3, + "median": 89.3, + "p95": 89.3, + "max": 89.3 + } + } +} diff --git a/docs/tuning/2026-08-11-monitor-cpu/probe.csv b/docs/tuning/2026-08-11-monitor-cpu/probe.csv new file mode 100644 index 0000000..4e17aab --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/probe.csv @@ -0,0 +1,121 @@ +t_s,cpu_probe_floor_cpu_pct,cpu_probe_raw_cpu_pct,cpu_probe_deser_cpu_pct,health_monitor_cpu_pct,task_server_cpu_pct,cpu_probe_floor_rss_mb,cpu_probe_raw_rss_mb,cpu_probe_deser_rss_mb,health_monitor_rss_mb,task_server_rss_mb,load1 +1.0,0.0,8.0,9.0,15.0,6.0,63.7,68.1,70.1,77.9,89.0,5.63 +2.0,1.0,7.99,7.99,14.99,5.99,63.7,68.1,70.1,77.9,89.0,5.63 +3.0,0.0,7.0,8.99,13.99,6.0,63.7,68.1,70.1,77.9,89.0,5.17 +4.0,1.0,8.0,8.0,15.99,6.0,63.7,68.1,70.1,77.9,89.0,5.17 +5.0,0.0,7.99,9.99,16.99,7.99,63.7,68.1,70.1,77.9,89.0,5.17 +6.0,0.0,8.99,9.99,18.98,5.99,63.7,68.1,70.1,77.9,89.0,5.17 +7.0,0.0,8.99,9.99,17.99,7.99,63.7,68.1,70.1,77.9,89.0,5.17 +8.0,1.0,7.99,9.99,17.99,6.99,63.7,68.1,70.1,77.9,89.1,5.16 +9.0,0.0,8.98,8.98,19.97,6.99,63.7,68.1,69.2,77.9,89.1,5.16 +10.0,0.0,8.99,10.99,16.98,6.99,63.7,68.1,69.2,77.9,89.1,5.16 +11.0,1.0,8.98,9.98,15.96,7.98,63.7,68.1,69.2,78.0,89.1,5.16 +12.0,0.0,7.98,9.98,16.96,6.98,63.7,68.1,69.2,78.0,89.1,5.16 +13.0,2.0,8.99,9.99,16.98,7.99,63.7,68.1,69.2,78.0,89.1,4.83 +14.0,0.0,7.99,9.99,16.99,6.99,63.7,68.1,69.2,78.0,89.1,4.83 +15.0,0.0,8.99,9.99,17.99,6.99,63.7,68.1,69.2,78.0,89.1,4.83 +16.0,0.0,7.99,8.99,16.99,7.0,63.7,68.1,69.2,78.0,89.1,4.83 +17.0,0.0,8.99,9.99,16.99,6.99,63.7,68.1,69.2,78.0,89.1,4.83 +18.0,1.0,7.99,8.99,18.99,8.99,63.7,68.1,69.3,78.0,89.1,5.88 +19.0,1.0,8.99,9.99,16.99,6.0,63.7,68.1,69.3,78.0,89.1,5.88 +20.0,0.0,7.99,8.99,16.98,6.99,63.7,68.1,69.3,78.0,89.1,5.88 +21.0,0.0,8.99,10.99,17.99,7.0,63.7,68.1,69.3,78.0,89.1,5.88 +22.0,1.0,7.99,8.99,16.99,7.0,63.7,68.1,69.3,78.0,89.1,5.88 +23.0,0.0,7.99,9.99,18.98,6.99,63.7,68.1,69.3,78.0,89.1,5.41 +24.0,0.0,8.99,8.99,16.99,7.99,63.7,68.1,69.3,78.0,89.1,5.41 +25.0,1.0,8.99,9.99,17.99,7.0,63.7,68.2,69.3,78.0,89.1,5.41 +26.0,1.0,8.99,9.99,16.99,6.99,63.7,68.2,69.3,78.0,89.1,5.41 +27.0,0.0,8.99,9.99,18.99,6.99,63.7,68.2,69.3,78.0,89.1,5.41 +28.0,0.0,7.98,9.98,17.96,6.98,63.7,68.2,69.3,78.0,89.1,5.78 +29.0,1.0,8.99,9.99,17.99,7.99,63.7,68.2,69.3,78.0,89.1,5.78 +30.0,0.0,8.99,9.99,17.99,6.0,63.7,68.2,69.3,78.0,89.1,5.78 +31.0,1.0,8.99,9.99,17.99,6.99,63.7,68.2,69.3,78.0,89.1,5.78 +32.0,0.0,7.0,7.0,14.99,6.0,63.7,68.2,69.3,78.0,89.1,5.78 +33.0,0.0,6.99,9.99,14.99,6.0,63.7,68.2,69.3,78.0,89.1,5.32 +34.0,1.0,8.99,7.99,14.98,5.99,63.8,68.2,69.3,78.0,89.1,5.32 +35.0,0.0,6.98,8.98,14.96,5.99,63.8,68.2,69.3,78.0,89.1,5.32 +36.0,0.0,7.98,8.98,15.96,5.99,63.8,68.2,69.3,78.0,89.1,5.32 +37.0,1.0,8.99,9.99,16.98,6.99,63.8,68.2,69.3,78.0,89.1,5.32 +38.0,0.0,7.99,8.99,17.98,6.99,63.8,68.2,69.3,78.0,89.1,5.05 +39.0,0.0,7.99,9.99,17.99,6.99,63.8,68.2,69.3,78.0,89.1,5.05 +40.0,1.0,8.99,9.99,16.99,6.99,63.8,68.2,69.3,78.0,89.1,5.05 +41.0,0.0,8.99,9.99,16.99,7.99,63.8,68.2,69.3,78.0,89.1,5.05 +42.0,1.0,8.99,9.99,19.98,7.99,63.8,68.2,69.3,78.1,89.1,5.05 +43.0,1.0,7.99,9.98,16.97,6.99,63.8,68.2,69.3,78.1,89.1,4.73 +44.0,0.0,9.99,9.99,18.99,6.99,63.8,68.2,69.3,78.1,89.1,4.73 +45.0,0.0,8.99,9.99,17.99,6.99,63.8,68.2,69.3,78.1,89.1,4.73 +46.0,0.0,7.99,9.98,17.97,6.99,63.8,68.2,69.3,78.1,89.1,4.73 +47.0,0.0,8.99,9.99,16.98,6.99,63.8,68.2,69.3,78.1,89.1,4.73 +48.1,1.0,8.97,9.96,17.93,6.97,63.8,68.2,69.3,78.1,89.1,4.91 +49.1,1.0,8.99,9.99,17.98,5.99,63.8,68.2,69.3,78.1,89.1,4.91 +50.1,0.0,8.99,9.99,17.98,7.99,63.8,68.2,69.4,78.1,89.1,4.91 +51.1,0.0,7.99,8.99,14.99,6.0,63.8,68.2,69.4,78.1,89.1,4.91 +52.1,2.0,7.99,7.99,13.99,6.0,63.8,68.2,69.4,78.1,89.1,4.91 +53.1,0.0,6.97,7.96,15.93,5.97,63.8,68.2,69.4,78.1,89.1,4.52 +54.1,0.0,7.0,9.99,13.99,6.0,63.8,68.2,69.4,78.1,89.1,4.52 +55.1,0.0,7.0,7.99,14.99,7.0,63.8,68.2,69.4,78.1,89.1,4.52 +56.1,1.0,8.99,8.99,17.99,6.99,63.8,68.2,69.4,78.1,89.1,4.52 +57.1,0.0,8.95,9.94,15.91,6.96,63.8,68.3,69.4,78.1,89.1,4.52 +58.1,0.0,7.99,8.99,17.98,7.99,63.8,68.3,69.4,78.1,89.1,4.15 +59.1,1.0,8.99,10.99,17.99,7.0,63.8,68.3,69.4,78.1,89.1,4.15 +60.1,0.0,8.99,8.99,17.99,6.99,63.8,68.3,69.4,78.1,89.1,4.15 +61.1,1.0,7.99,10.99,17.99,7.0,63.8,68.3,69.4,78.1,89.1,4.15 +62.1,0.0,8.99,8.99,17.99,7.99,63.8,68.3,69.4,78.1,89.1,4.15 +63.1,1.0,8.99,8.99,17.98,6.99,63.8,68.3,69.4,78.1,89.1,3.9 +64.1,0.0,8.99,10.99,16.99,6.99,63.8,68.3,69.4,78.1,89.1,3.9 +65.1,0.0,8.99,8.99,16.99,7.99,63.8,68.3,69.4,78.1,89.1,3.9 +66.1,1.0,7.99,9.99,17.99,7.0,63.8,68.3,69.4,78.1,89.1,3.9 +67.1,0.0,8.99,10.99,19.99,6.0,63.8,68.3,69.4,78.1,89.1,3.9 +68.1,0.0,8.99,9.99,18.99,6.99,63.8,68.3,69.4,78.1,89.1,3.75 +69.1,0.0,8.99,8.99,17.99,6.99,63.8,68.3,69.4,78.1,89.1,3.75 +70.1,1.0,7.99,10.99,16.98,5.99,63.8,68.3,69.4,78.1,89.1,3.75 +71.1,1.0,9.99,9.99,16.99,7.99,63.8,68.3,69.4,78.1,89.1,3.75 +72.1,0.0,8.99,9.99,16.99,7.0,63.8,68.3,69.4,78.1,89.1,3.75 +73.1,1.0,8.99,9.99,17.98,6.99,63.8,68.3,69.4,78.1,89.1,3.45 +74.1,0.0,7.99,9.99,17.98,6.99,63.9,68.3,69.4,78.1,89.1,3.45 +75.1,0.0,9.99,10.99,17.99,6.99,63.9,68.3,69.4,78.1,89.1,3.45 +76.1,1.0,7.99,8.99,17.99,7.99,63.9,68.3,69.4,78.1,89.1,3.45 +77.1,1.0,8.99,9.99,16.99,6.0,63.9,68.3,69.4,78.1,89.1,3.45 +78.1,0.0,8.99,9.99,16.99,6.99,63.9,68.3,69.4,78.1,89.1,3.17 +79.1,0.0,7.99,9.99,17.98,6.99,63.9,68.3,69.4,78.1,89.1,3.17 +80.1,0.0,8.98,9.98,17.96,6.99,63.9,68.3,69.5,78.2,89.1,3.17 +81.1,1.0,9.99,10.99,16.99,7.99,63.9,68.3,69.5,78.2,89.1,3.17 +82.1,1.0,8.99,8.99,16.98,6.99,63.9,68.3,69.5,78.2,89.1,3.17 +83.1,0.0,7.99,10.99,16.98,7.99,63.9,68.3,69.5,78.2,89.1,2.92 +84.1,0.0,7.99,8.99,17.98,5.99,63.9,68.3,69.5,78.2,89.1,2.92 +85.1,1.0,8.99,9.99,17.99,6.99,63.9,68.3,69.5,78.2,89.1,2.92 +86.1,0.0,7.99,9.98,16.97,6.99,63.9,68.3,69.5,78.2,89.1,2.92 +87.1,1.0,7.99,9.99,18.99,6.0,63.9,68.3,69.5,78.2,89.1,2.92 +88.1,0.0,8.99,8.99,16.99,7.99,63.9,68.4,69.5,78.2,89.1,2.76 +89.1,0.0,8.98,10.98,17.96,6.99,63.9,68.4,69.5,78.2,89.1,2.76 +90.1,1.0,8.97,9.97,18.94,6.98,63.9,68.4,69.5,78.2,89.1,2.76 +91.1,0.0,7.99,9.99,17.99,7.99,63.9,68.4,69.5,78.2,89.1,2.76 +92.1,1.0,8.99,9.99,17.99,7.0,63.9,68.4,69.5,78.2,89.1,2.76 +93.1,0.0,8.99,9.99,16.99,6.0,63.9,68.4,69.5,78.2,89.1,3.98 +94.1,0.0,7.99,9.99,17.98,7.99,63.9,68.4,69.5,78.2,89.1,3.98 +95.1,0.0,8.98,9.98,16.97,5.99,63.9,68.4,69.5,78.2,89.1,3.98 +96.1,1.0,7.98,9.98,17.96,6.98,63.9,68.4,69.5,78.2,89.1,3.98 +97.1,0.0,8.99,9.99,16.99,6.99,63.9,68.4,69.5,78.2,89.1,3.98 +98.1,1.0,9.98,9.98,17.97,6.99,63.9,68.4,69.5,78.2,89.1,3.82 +99.1,1.0,8.99,9.99,15.99,6.99,63.9,68.4,69.5,78.2,89.1,3.82 +100.1,0.0,7.99,9.99,16.99,6.99,63.9,68.4,69.5,78.2,89.1,3.82 +101.1,0.0,8.99,8.99,16.98,6.99,63.9,68.4,69.5,78.2,89.1,3.82 +102.1,1.0,7.99,9.99,16.98,6.99,63.9,68.4,69.5,78.2,89.1,3.82 +103.1,0.0,7.99,9.99,17.99,6.99,63.9,68.4,69.5,78.2,89.1,3.52 +104.1,0.0,8.99,9.99,16.99,6.0,63.9,68.4,69.5,78.2,89.1,3.52 +105.1,1.0,8.98,9.98,18.97,7.99,63.9,68.4,69.5,78.2,89.1,3.52 +106.1,0.0,7.98,8.97,16.95,7.98,64.0,68.4,69.5,78.2,89.1,3.52 +107.1,1.0,8.98,10.98,17.97,6.99,64.0,68.4,69.5,78.2,89.1,3.52 +108.1,0.0,8.99,8.99,16.98,6.99,64.0,68.4,69.5,78.2,89.1,3.24 +109.1,1.0,7.99,9.99,17.98,6.99,64.0,68.4,69.5,78.2,89.1,3.24 +110.1,0.0,7.99,9.99,17.99,7.0,64.0,68.4,69.5,78.2,89.1,3.24 +111.1,0.0,8.99,9.99,16.98,7.99,64.0,68.4,69.5,78.2,89.1,3.24 +112.1,1.0,9.97,9.97,18.94,7.98,64.0,68.4,69.5,78.2,89.1,3.24 +113.1,0.0,8.99,9.99,17.99,6.0,64.0,68.4,69.5,78.2,89.1,3.06 +114.1,1.0,7.99,8.99,17.99,7.99,64.0,68.4,69.6,78.2,89.1,3.06 +115.1,0.0,7.99,10.99,17.99,6.99,64.0,68.4,69.6,78.2,89.1,3.06 +116.1,0.0,9.99,9.99,17.99,7.0,64.0,68.4,69.6,78.2,89.1,3.06 +117.1,0.0,7.99,9.99,17.98,6.99,64.0,68.5,69.6,78.2,89.1,3.06 +118.1,1.0,8.99,8.99,15.99,6.99,64.0,68.5,69.6,78.2,89.1,2.89 +119.1,1.0,8.99,9.99,16.99,7.99,64.0,68.5,69.6,78.2,89.1,2.89 +120.1,0.0,8.99,9.99,17.99,6.0,64.0,68.5,69.6,78.2,89.1,2.89 diff --git a/docs/tuning/2026-08-11-monitor-cpu/probe.json b/docs/tuning/2026-08-11-monitor-cpu/probe.json new file mode 100644 index 0000000..5bf9eae --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/probe.json @@ -0,0 +1,75 @@ +{ + "csv": "/home/michael/tmp/monitor-cpu/probe/probe.csv", + "samples": 120, + "span_s": 119.1, + "load1": { + "mean": 4.3, + "median": 4.2, + "p95": 5.8, + "max": 5.9 + }, + "cpu_pct": { + "cpu_probe_floor": { + "mean": 0.4, + "median": 0.0, + "p95": 1.0, + "max": 2.0 + }, + "cpu_probe_raw": { + "mean": 8.6, + "median": 9.0, + "p95": 10.0, + "max": 10.0 + }, + "cpu_probe_deser": { + "mean": 9.7, + "median": 10.0, + "p95": 11.0, + "max": 11.0 + }, + "health_monitor": { + "mean": 17.4, + "median": 18.0, + "p95": 19.0, + "max": 20.0 + }, + "task_server": { + "mean": 7.0, + "median": 7.0, + "p95": 8.0, + "max": 9.0 + } + }, + "rss_mb": { + "cpu_probe_floor": { + "mean": 63.8, + "median": 63.8, + "p95": 64.0, + "max": 64.0 + }, + "cpu_probe_raw": { + "mean": 68.3, + "median": 68.3, + "p95": 68.4, + "max": 68.5 + }, + "cpu_probe_deser": { + "mean": 69.4, + "median": 69.4, + "p95": 70.1, + "max": 70.1 + }, + "health_monitor": { + "mean": 78.1, + "median": 78.1, + "p95": 78.2, + "max": 78.2 + }, + "task_server": { + "mean": 89.1, + "median": 89.1, + "p95": 89.1, + "max": 89.1 + } + } +} diff --git a/docs/tuning/2026-08-11-monitor-cpu/runs.txt b/docs/tuning/2026-08-11-monitor-cpu/runs.txt new file mode 100644 index 0000000..059b72b --- /dev/null +++ b/docs/tuning/2026-08-11-monitor-cpu/runs.txt @@ -0,0 +1,37 @@ +Raw sampler output behind docs/tuning/2026-08-11-monitor-cpu.md. + +All runs: mote-01 (Raspberry Pi 5, 4 cores), `pixi run robot` + `pixi run tasks` +up, no task dispatched, 1 s sampling interval. Each .csv is one column per +node per sample plus load1; .json is its summary (mean/median/p95/max). + +Re-summarize any of them without a robot: + + python -m mote_bringup.tools.node_cpu --summary .csv + + before-idle.csv 60 s. Stack as found: the drive servos were not + answering, so /joint_states was 1.6 Hz and /tf 33 Hz. + Kept because it is the state the robot was actually in, + not because it is comparable with anything. + + before-joints50.csv 60 s. Same stack, with a synthetic 50 Hz /joint_states + standing in for a healthy control loop. + + after-joints50.csv 60 s. The patched build, same condition — except the + servo bus had partly recovered between the two runs + (3159 read failures vs 564, /tf 33 Hz vs 51 Hz), which + is why this pair is reported as a warning about + sequential measurement rather than as a result. + + ab-patched-vs-unpatched.csv + 120 s. The result. Patched and unpatched builds of both + nodes running at the same instant, the unpatched ones + renamed `_b`, so the two see identical input: + health_monitor 17.1 vs health_monitor_b 18.1, + task_server 6.9 vs task_server_b 7.8. + + probe.csv 120 s. cpu_probe floor/raw/deser: what a bare rclpy + node costs, what four subscriptions add, and what + deserializing them adds on top. + + probe-tf.csv 90 s. The same three probes plus one holding only a + TransformListener. This is the table quoted in the doc. diff --git a/mkdocs.yml b/mkdocs.yml index 3333925..d2153bd 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -155,3 +155,4 @@ nav: - " ↳ gated benchmark": tuning/2026-07-28-icp-velocity-gate/bench-gated.md - Slip detection: tuning/2026-07-28-slip-detection.md - Camera layer decay: tuning/2026-07-29-camera-layer-decay.md + - Monitor-node CPU: tuning/2026-08-11-monitor-cpu.md diff --git a/mote_bringup/mote_bringup/health_monitor.py b/mote_bringup/mote_bringup/health_monitor.py index 37d1cae..57169ae 100644 --- a/mote_bringup/mote_bringup/health_monitor.py +++ b/mote_bringup/mote_bringup/health_monitor.py @@ -102,6 +102,7 @@ def __init__(self, spec): self.count = 0 def on_msg(self, _msg): + """Record an arrival. The payload is never read, and arrives raw.""" self.last_stamp = time.monotonic() self.count += 1 @@ -157,7 +158,16 @@ def __init__(self): for spec in cfg.get("topics", []): watch = _TopicWatch(spec) msg_type = get_message(spec["type"]) - self.create_subscription(msg_type, spec["topic"], watch.on_msg, 10) + # Subscribed raw: the watch only counts and timestamps, so the + # callback never touches a field. These are the robot's highest-rate + # topics — a 50 Hz JointState, two LaserScans and a ~30 fps + # CompressedImage — and deserializing each one into a Python object + # to learn that it arrived costs more than everything this node does + # with it. The type is still needed: it is what selects the + # typesupport, only the delivered object changes (bytes). + self.create_subscription( + msg_type, spec["topic"], watch.on_msg, 10, raw=True + ) self.topics.append(watch) self.tf_watches = [_TfWatch(s) for s in cfg.get("tf", [])] @@ -171,6 +181,9 @@ def __init__(self): ) self.forwarded = {} if cfg.get("subscribe_diagnostics", True): + # Deserialized, unlike the freshness watches above: this callback + # reads status.name and status.level. It is also the one low-rate + # subscription here, published once per aggregation period. self.create_subscription( DiagnosticArray, "diagnostics", self._on_diagnostics, 10 ) diff --git a/mote_bringup/test/test_health_monitor_node.py b/mote_bringup/test/test_health_monitor_node.py new file mode 100644 index 0000000..0aefb95 --- /dev/null +++ b/mote_bringup/test/test_health_monitor_node.py @@ -0,0 +1,138 @@ +"""The health monitor as a running node: raw subscriptions still report. + +``test_health_monitor.py`` covers the roll-up decisions as plain function calls. +What it cannot cover is the delivery underneath them. The watched topics are +subscribed raw — the callback only counts arrivals, so nothing is gained by +building a Python message first — and ``raw=True`` is exactly the kind of change +that fails silently: a subscription that delivers nothing leaves a node which +still runs, still publishes on time, and reports every subsystem as missing. So +this drives the real node with a real publisher and asserts both halves: that +the topics are subscribed raw, and that a raw arrival still reaches the summary. +""" + +import os +import random + +# A stray ROS_DOMAIN_ID here would put this test on the same graph as a real +# robot. Claim an unused domain and stay on localhost before rclpy is imported. +os.environ["ROS_DOMAIN_ID"] = str(random.randint(64, 200)) +os.environ["ROS_AUTOMATIC_DISCOVERY_RANGE"] = "LOCALHOST" + +import pytest # noqa: E402 +import rclpy # noqa: E402 +from diagnostic_msgs.msg import DiagnosticArray, DiagnosticStatus # noqa: E402 +from rclpy.executors import SingleThreadedExecutor # noqa: E402 +from rclpy.node import Node # noqa: E402 +from sensor_msgs.msg import LaserScan # noqa: E402 +from std_msgs.msg import String # noqa: E402 + +from mote_bringup.health_monitor import HealthMonitor # noqa: E402 + +CONFIG = """\ +period: 0.2 +topics: + - name: scan + topic: /scan + type: sensor_msgs/msg/LaserScan + min_rate: 5.0 + timeout: 2.0 + severity: critical +tf: [] +subscribe_diagnostics: true +""" + +SCAN_RATE = 20.0 + + +class _Lidar(Node): + """Publishes scans at a rate comfortably above the configured floor.""" + + def __init__(self): + super().__init__("fake_lidar") + self.published = 0 + self.pub = self.create_publisher(LaserScan, "/scan", 10) + self.create_timer(1.0 / SCAN_RATE, self._publish) + + def _publish(self): + msg = LaserScan() + msg.header.stamp = self.get_clock().now().to_msg() + msg.header.frame_id = "laser" + msg.ranges = [1.0] * 360 + self.pub.publish(msg) + self.published += 1 + + +class _Listener(Node): + def __init__(self): + super().__init__("health_listener") + self.summaries = [] + self.aggregates = [] + self.create_subscription(String, "health", self._on_health, 10) + self.create_subscription( + DiagnosticArray, "diagnostics_agg", self.aggregates.append, 10 + ) + + def _on_health(self, msg): + self.summaries.append(msg.data) + + +@pytest.fixture +def monitor(tmp_path, monkeypatch): + monkeypatch.setenv("MOTE_HOME", str(tmp_path)) + (tmp_path / "health.yaml").write_text(CONFIG) + rclpy.init() + node = HealthMonitor() + yield node + node.destroy_node() + rclpy.try_shutdown() + + +def _spin(nodes, seconds): + executor = SingleThreadedExecutor() + for node in nodes: + executor.add_node(node) + deadline = nodes[0].get_clock().now().nanoseconds / 1e9 + seconds + while nodes[0].get_clock().now().nanoseconds / 1e9 < deadline: + executor.spin_once(timeout_sec=0.02) + + +def _subscriptions(node): + return {sub.topic_name.lstrip("/"): sub for sub in node.subscriptions} + + +def test_watched_topics_are_subscribed_raw(monitor): + """Freshness watches take bytes; the roll-up's own input does not. + + The /diagnostics subscription reads status.name and status.level, so it is + the one that must stay deserialized — subscribing it raw would hand the + callback bytes it would index as a message, and the forwarded statuses + would vanish from the summary rather than raise. + """ + subs = _subscriptions(monitor) + assert subs["scan"].raw is True + assert subs["diagnostics"].raw is False + + +def test_a_raw_scan_still_reaches_the_summary(monitor): + """A counted arrival is worth nothing if the count never moves.""" + lidar = _Lidar() + listener = _Listener() + try: + _spin([monitor, lidar, listener], seconds=2.0) + finally: + lidar.destroy_node() + listener.destroy_node() + + assert lidar.published > 10, "the publisher itself never ran" + watch = monitor.topics[0] + assert watch.count or watch.last_stamp, "no raw message ever arrived" + + assert listener.summaries, "nothing published on /health" + assert listener.summaries[-1] == "OK", listener.summaries + # The aggregate keeps its shape: the mote roll-up first, then one status per + # subsystem, and the measured rate is a real one rather than zero. + last = listener.aggregates[-1] + assert [s.name for s in last.status] == ["mote", "scan"] + assert last.status[0].level == DiagnosticStatus.OK + values = {kv.key: kv.value for kv in last.status[1].values} + assert float(values["rate_hz"]) >= 5.0, values diff --git a/mote_bringup/tools/node_cpu.py b/mote_bringup/tools/node_cpu.py new file mode 100755 index 0000000..8f15f2d --- /dev/null +++ b/mote_bringup/tools/node_cpu.py @@ -0,0 +1,266 @@ +#!/usr/bin/env python3 +"""Sample per-node CPU on the robot, one column per node. + +The monitor nodes are Python processes whose own logic runs at 1-10 Hz, so what +they cost is dominated by how often they are *woken* rather than by what they +compute. Answering that needs a figure per node, before and after a change, on +the same workload: + + python -m mote_bringup.tools.node_cpu --duration 60 --tag idle \ + --out docs/tuning/2026-08-11-monitor-cpu/before + +CPU comes from ``/proc//stat`` as a delta of utime+stime over the interval, +the same source and arithmetic as the benchmark's ``overhead.py`` — which is not +reused here because it lives in ``mote_simulation``, deliberately excluded from +``pixi run sync``, and this has to run on the Pi. It also totals its matches into +one figure, where the question here is which node is expensive. Both are stdlib +only: ``pidstat`` is installed on neither the workstation nor the Pi. + +A Python node has no distinguishing executable — every one of them is the +interpreter — so a node is identified by what its command line says it is +running (see :func:`node_instance`), confined to processes whose command line +names ``--prefix`` (default: this checkout). A box running several worktrees, or +holding a stack leaked by a dead agent job (see ``pixi run sweep``), would +otherwise have another checkout's monitors counted as this one's. + +Two builds of one node can be sampled *against each other* by starting the +second one renamed, which is the only sound way to compare them on a robot whose +own condition drifts between runs — a servo bus that answers in one run and not +the next moves ``/joint_states`` and ``/tf`` by tens of Hz, and those rates are +exactly what the monitors cost: + + ros2 run mote_bringup health_monitor --ros-args -r __node:=health_monitor_b \ + -r diagnostics_agg:=diagnostics_agg_b -r health:=health_b + python -m mote_bringup.tools.node_cpu --nodes health_monitor,health_monitor_b + +Load average is recorded beside each sample: a percentage of a core means little +without knowing how contended the machine was when it was measured. +""" + +from __future__ import annotations + +import argparse +import csv +import json +import os +import statistics +import sys +import time +from pathlib import Path + +# The nodes this exists to weigh: the monitors, plus the task layer. +DEFAULT_NODES = ["health_monitor", "task_server", "slip_monitor", "system_monitor"] + +CLOCK_TICKS = os.sysconf("SC_CLK_TCK") +PAGE_SIZE = os.sysconf("SC_PAGE_SIZE") +REPO = Path(__file__).resolve().parents[2] + + +def _candidates(prefix): + """[(pid, argv)] for the processes this sampler is allowed to match.""" + self_pid = os.getpid() + this_file = Path(__file__).name + out = [] + for entry in Path("/proc").iterdir(): + if not entry.name.isdigit(): + continue + pid = int(entry.name) + if pid == self_pid: + continue + try: + raw = entry.joinpath("cmdline").read_bytes().decode(errors="replace") + except OSError: + continue + argv = [tok for tok in raw.split("\0") if tok] + if not argv: + continue + joined = " ".join(argv) + # This sampler names every node it measures on its own command line. + if this_file in joined or "node_cpu" in joined: + continue + if prefix and prefix not in joined: + continue + out.append((pid, argv)) + return out + + +def node_instance(argv): + """The node this process *is*, or None if it is not one. + + A node started through ``ros2 run`` or ``pixi run`` is wrapped by processes + that repeat its whole command line in their own, so a substring match would + weigh a wrapper — one of which does no work at all — instead of the node. + The node is the only one of them an interpreter runs directly: argv[0] is a + python and argv[1] is the installed entry point, whose name is the node's. + + The name reported is the ``__node:=`` rename where a launch or a remap gave + it one, and the entry point's own name otherwise. That is what lets two + builds of one node be weighed against each other in a single run: they + differ only by the rename, and the rename is what tells them apart. + """ + if len(argv) < 2 or "python" not in os.path.basename(argv[0]): + return None + program = os.path.basename(argv[1]) + if program == "ros2": + return None + for tok in argv[2:]: + if tok.startswith("__node:="): + return tok.split(":=", 1)[1] + return program + + +def matching_pids(nodes, prefix): + """{node name: pid} for the named nodes running out of ``prefix``. + + A node that is not running is simply absent, so a run with the stack half up + still yields figures for what is up. + """ + found = {} + for pid, argv in _candidates(prefix): + instance = node_instance(argv) + if instance in nodes and instance not in found: + found[instance] = pid + return found + + +def proc_cpu_ticks(pid): + """utime+stime in clock ticks, or None if the process is gone.""" + try: + # comm can contain spaces and parentheses, so split after the last ')'. + raw = Path(f"/proc/{pid}/stat").read_text() + fields = raw[raw.rindex(")") + 2 :].split() + except (OSError, ValueError): + return None + # fields[0] is state (field 3 in proc(5)); utime/stime are fields 14/15. + return int(fields[11]) + int(fields[12]) + + +def proc_rss_bytes(pid): + try: + return int(Path(f"/proc/{pid}/statm").read_text().split()[1]) * PAGE_SIZE + except (OSError, IndexError, ValueError): + return 0 + + +def run(args): + nodes = [n for n in args.nodes.split(",") if n] + out_dir = Path(args.out) + out_dir.mkdir(parents=True, exist_ok=True) + csv_path = out_dir / f"{args.tag}.csv" + + pids = matching_pids(nodes, args.prefix) + missing = [n for n in nodes if n not in pids] + if missing: + print(f"not running: {', '.join(missing)}", file=sys.stderr) + if not pids: + print("no matching processes — is the stack up?", file=sys.stderr) + return 1 + print( + f"sampling {args.duration:.0f}s: " + + ", ".join(f"{n}={p}" for n, p in pids.items()) + ) + + prev = {n: proc_cpu_ticks(p) for n, p in pids.items()} + prev_t = time.monotonic() + deadline = prev_t + args.duration + start = prev_t + + with open(csv_path, "w", newline="") as f: + w = csv.writer(f) + w.writerow( + [ + "t_s", + *(f"{n}_cpu_pct" for n in nodes), + *(f"{n}_rss_mb" for n in nodes), + "load1", + ] + ) + while time.monotonic() < deadline: + time.sleep(args.interval) + now = time.monotonic() + dt = now - prev_t + cpu, rss = {}, {} + for node, pid in pids.items(): + ticks = proc_cpu_ticks(pid) + # A node that died mid-run leaves blanks rather than a huge + # negative delta from a recycled pid. + if ticks is None or prev.get(node) is None: + cpu[node] = rss[node] = None + else: + cpu[node] = 100.0 * (ticks - prev[node]) / CLOCK_TICKS / dt + rss[node] = proc_rss_bytes(pid) / 1e6 + prev[node] = ticks + w.writerow( + [ + round(now - start, 1), + *(None if cpu.get(n) is None else round(cpu[n], 2) for n in nodes), + *(None if rss.get(n) is None else round(rss[n], 1) for n in nodes), + round(os.getloadavg()[0], 2), + ] + ) + f.flush() + prev_t = now + return summarize(csv_path, nodes) + + +def summarize(csv_path, nodes=None): + with open(csv_path) as f: + rows = list(csv.DictReader(f)) + if not rows: + print(f"{csv_path}: no samples", file=sys.stderr) + return 1 + if nodes is None: + nodes = [k[: -len("_cpu_pct")] for k in rows[0] if k.endswith("_cpu_pct")] + + def stat(key): + vals = [float(r[key]) for r in rows if r.get(key) not in (None, "")] + if not vals: + return None + return { + "mean": round(statistics.mean(vals), 1), + "median": round(statistics.median(vals), 1), + "p95": round(sorted(vals)[min(len(vals) - 1, int(0.95 * len(vals)))], 1), + "max": round(max(vals), 1), + } + + result = { + "csv": str(csv_path), + "samples": len(rows), + "span_s": round(float(rows[-1]["t_s"]) - float(rows[0]["t_s"]), 1), + "load1": stat("load1"), + "cpu_pct": {n: stat(f"{n}_cpu_pct") for n in nodes}, + "rss_mb": {n: stat(f"{n}_rss_mb") for n in nodes}, + } + result["cpu_pct"] = {k: v for k, v in result["cpu_pct"].items() if v} + result["rss_mb"] = {k: v for k, v in result["rss_mb"].items() if v} + Path(csv_path).with_suffix(".json").write_text(json.dumps(result, indent=2)) + print(json.dumps(result, indent=2)) + return 0 + + +def main(): + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--duration", type=float, default=60.0, help="wall s to sample") + ap.add_argument("--interval", type=float, default=1.0, help="s between samples") + ap.add_argument("--out", default="node_cpu", help="directory for the CSV/JSON") + ap.add_argument("--tag", default="run", help="names the CSV, e.g. 'before-idle'") + ap.add_argument( + "--nodes", + default=",".join(DEFAULT_NODES), + help=f"comma-separated node names to match (default: {','.join(DEFAULT_NODES)})", + ) + ap.add_argument( + "--prefix", + default=str(REPO), + help="only count processes whose command line names this path " + "(default: this checkout, so a sibling worktree's stack is ignored)", + ) + ap.add_argument("--summary", default="", help="skip sampling; summarize this CSV") + args = ap.parse_args() + if args.summary: + return summarize(Path(args.summary)) + return run(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/mote_tasks/mote_tasks/task_server.py b/mote_tasks/mote_tasks/task_server.py index ff468bf..a95bf65 100644 --- a/mote_tasks/mote_tasks/task_server.py +++ b/mote_tasks/mote_tasks/task_server.py @@ -12,6 +12,11 @@ ``zones_file``, falling back to the committed config/zones.default.yaml). Outcomes are published on ``task/status`` as accepted/rejected/succeeded/failed strings. + +The active tree is ticked at ``tick_period`` while a task is running and at the +slower ``idle_tick_period`` between them, since a tree waiting for work has +nothing to advance. Command acceptance is unaffected either way: it happens in +the subscription callback, not on a tick. """ import os @@ -32,6 +37,7 @@ def __init__(self, **node_kwargs): super().__init__("task_server", **node_kwargs) zones_file = self.declare_parameter("zones_file", "").value tick_period = self.declare_parameter("tick_period", 0.1).value + idle_tick_period = self.declare_parameter("idle_tick_period", 1.0).value pick_duration = self.declare_parameter("pick_duration", 3.0).value place_duration = self.declare_parameter("place_duration", 3.0).value @@ -67,7 +73,35 @@ def __init__(self, **node_kwargs): self.status_pub = self.create_publisher(String, "task/status", 1) self.create_subscription(String, "task/command", self.on_command, 1) self.last_tip = None - self.create_timer(tick_period, self.tick) + # Two rates, because a tree between missions has nothing to advance: it + # idles in WaitForTask, whose whole update() is one blackboard read, and + # ticking that at the mission rate is ten wake-ups a second to learn + # nothing has changed. An idle rate faster than the mission rate would + # be a contradiction, so it is floored at it. + self.tick_period = tick_period + self.idle_tick_period = max(idle_tick_period, tick_period) + self.ticking_fast = False + self.tick_timer = self.create_timer(self.idle_tick_period, self.tick) + + def _set_tick_rate(self, active: bool): + """Tick at the mission rate while a task runs, slowly between them. + + The timer is *reset* as well as re-periodded, because setting a period + does not move the expiry already pending — measured: a 5 s timer 0.3 s + into its period still reports 4.7 s to go after its period is set to + 0.05 s, and 0.05 s after ``reset()``. Without the reset, a command + accepted just after an idle tick would wait out the rest of the idle + period before the tree ticked at all. Accepting a command is already + independent of the tick (``on_command`` publishes the outcome itself); + what this protects is the first tick of the accepted tree, which is + what sends the Nav2 goal and starts the robot driving. + """ + if active == self.ticking_fast: + return + self.ticking_fast = active + period = self.tick_period if active else self.idle_tick_period + self.tick_timer.timer_period_ns = int(period * 1e9) + self.tick_timer.reset() def publish_status(self, text: str): self.get_logger().info(text) @@ -107,6 +141,7 @@ def on_command(self, msg: String): self.publish_status(f"rejected: '{msg.data}' ({e})") return self.blackboard.set(TASK_KEY, msg.data) + self._set_tick_rate(active=True) self.publish_status(f"accepted: {msg.data}") def tick(self): @@ -120,9 +155,11 @@ def tick(self): if root.status == py_trees.common.Status.SUCCESS: self.publish_status(f"succeeded: {self.blackboard.task}") self.blackboard.set(TASK_KEY, None) + self._set_tick_rate(active=False) elif root.status == py_trees.common.Status.FAILURE: self.publish_status(f"failed: {self.blackboard.task} (at {label})") self.blackboard.set(TASK_KEY, None) + self._set_tick_rate(active=False) def main(): diff --git a/mote_tasks/test/test_goto_tree.py b/mote_tasks/test/test_goto_tree.py index e10238f..031ef74 100644 --- a/mote_tasks/test/test_goto_tree.py +++ b/mote_tasks/test/test_goto_tree.py @@ -108,3 +108,84 @@ def test_goto_round_trip(ros, tmp_path): server.destroy_node() mock.destroy_node() + + +def test_idle_tick_rate_does_not_delay_the_mission(ros, tmp_path): + """The tree ticks slowly between missions, and instantly once given one. + + Between missions the tree ticks WaitForTask and nothing else, so it runs at + ``idle_tick_period``. The saving is only free if accepting a command + restores the mission rate *and* resets the timer, because setting a period + does not move the expiry already pending: an idling timer switched to the + mission rate still has the rest of its idle period to wait, and the first + tick of the accepted tree — the one that sends the Nav2 goal — is what + would wait. So the command is deliberately sent just *after* an idle tick, + with a whole idle period pending, which is the case the reset exists for. + """ + zones_file = tmp_path / "zones.yaml" + zones_file.write_text(ZONES) + server = TaskServer( + parameter_overrides=[ + Parameter("zones_file", value=str(zones_file)), + Parameter("tick_period", value=0.05), + Parameter("idle_tick_period", value=2.0), + ] + ) + mock = MockNav() + executor = SingleThreadedExecutor() + executor.add_node(server) + executor.add_node(mock) + + def period_s(): + return server.tick_timer.timer_period_ns / 1e9 + + def next_call_s(): + return server.tick_timer.time_until_next_call() / 1e9 + + assert period_s() == pytest.approx(2.0), "an idle tree is ticking at mission rate" + + assert spin_until( + executor, lambda: mock.command_pub.get_subscription_count() > 0 + ), "task_server never subscribed to task/command" + + # An idle tick has just fired, so a full idle period is pending. + assert spin_until(executor, lambda: next_call_s() > 1.5), "no idle tick fired" + + mock.command_pub.publish(String(data="goto kitchen")) + assert spin_until( + executor, lambda: any(s.startswith("accepted") for s in mock.statuses) + ), mock.statuses + accepted_at = time.monotonic() + assert period_s() == pytest.approx(0.05), "an accepted task is ticking at idle rate" + assert next_call_s() <= 0.1, ( + f"the accepted tree waits {next_call_s():.2f}s for its first tick — " + "the period changed but the pending idle expiry did not" + ) + + assert spin_until( + executor, lambda: any(s.startswith("succeeded") for s in mock.statuses) + ), mock.statuses + elapsed = time.monotonic() - accepted_at + assert elapsed < 1.5, f"the mission took {elapsed:.2f}s at the mission rate" + + # A finished mission hands the idle rate back, or the saving lasts one task. + assert period_s() == pytest.approx(2.0), "the tree kept ticking after the task" + + server.destroy_node() + mock.destroy_node() + + +def test_idle_rate_is_floored_at_the_mission_rate(ros, tmp_path): + """An idle rate faster than the mission rate is a contradiction, not a config.""" + zones_file = tmp_path / "zones.yaml" + zones_file.write_text(ZONES) + server = TaskServer( + parameter_overrides=[ + Parameter("zones_file", value=str(zones_file)), + Parameter("tick_period", value=0.5), + Parameter("idle_tick_period", value=0.1), + ] + ) + assert server.idle_tick_period == pytest.approx(0.5) + assert server.tick_timer.timer_period_ns / 1e9 == pytest.approx(0.5) + server.destroy_node() diff --git a/pixi.toml b/pixi.toml index f6547c6..f9934e2 100644 --- a/pixi.toml +++ b/pixi.toml @@ -130,6 +130,11 @@ chaos = "bash mote_bringup/test/chaos/chaos_restart.sh" # hardware). test_costmap_layers.py holds the settings this depends on; this is # what proves they still add up to a mark that expires. camera-decay-check = "python mote_bringup/tools/camera_layer_decay.py" +# Per-node CPU on a live robot, one column per node. Two builds of one node can +# be sampled against each other in a single run by starting the second renamed, +# which is the only sound way to compare them on hardware whose own condition +# drifts between runs. See docs/tuning/2026-08-11-monitor-cpu.md. +node-cpu = "python mote_bringup/tools/node_cpu.py" # SO-101 arm bench stack: the same ros2_control bring-up a mission uses, without # the lidar/camera/Nav2. During a mission the arm is already there (it lives in # mote_hardware, which owns the shared servo bus) — this is for bench work only.