feat(bench): add latency-under-load benchmark axis - #53
Conversation
Add bulk_produce helper tool, latload wrapper mode, harness integration in run_all.sh, and methodology specification for evaluating terminal DSR response latency while under heavy write load.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a40f3328bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case "$overlap" in | ||
| verified*) ;; | ||
| *) overlap_fail=$((overlap_fail + 1)) ;; |
There was a problem hiding this comment.
When the producer exits before the probe finishes—for example because a slow DSR run outlasts the fixed 8/20-second bulk window—this branch merely increments overlap_fail; the rep has already incremented got, contributed its samples to the pools, and contributed throughput to the median. The resulting pooled “under load” distribution can therefore include samples collected after the load ended, despite the failed containment check. Reject unverified reps from all aggregates or mark the result unmeasured.
Useful? React with 👍 / 👎.
| plmed="${1:-0}"; plp95="${2:-0}"; plp99="${3:-0}"; plmax="${4:-0}"; plcnt="${5:-0}" | ||
| mibps_med="$(printf "$mibps_samples" | median_f)" | ||
| degradation="$(awk -v l="$plmed" -v c="$pcmed" 'BEGIN{ if(c>0) printf "%.2f", l/c; else print "n/a" }')" | ||
| emit "$term" latency_under_load control pooled pooled_median_ns "$pcmed" ns |
There was a problem hiding this comment.
For --axes latency-under-load, these rows are written only to raw.tsv: aggregate.py constructs each supported axis explicitly and has no latency_under_load branch, and the visualization code likewise never consumes it. Consequently results.json omits the requested axis and table.md contains no latency-under-load section, even though run_all.sh advertises those as its machine- and human-readable outputs. Extend the aggregators when emitting this new axis.
Useful? React with 👍 / 👎.
| cat "$samples_noload" >> "$ctrl_pooled" 2>/dev/null | ||
| cat "$samples_load" >> "$load_pooled" 2>/dev/null |
There was a problem hiding this comment.
If one probe produces samples but the other returns zero replies, these unconditional appends retain the successful side even though the caller subsequently rejects the rep. Once any later rep succeeds, pooled_stats includes the rejected launch on only one side, so the reported control/load counts and degradation factor no longer represent paired launches. Delay both appends until both result counts and overlap have been validated.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds Axis 8 (
latency-under-load) to the terminal benchmark suite to evaluate DSR (Device Status Report) echo response latency while heavy write throughput is active on the same PTY.bench/tools/bulk_produce.cto generate bounded-duration write load against the PTY.bench/wrapper.shwith thelatloadmode to measure control vs under-load latency with progress overlap verification.latency-under-loadintobench/run_all.shharness and summary reporting.bench/METHODOLOGY.mdwith methodology and measurement specifications.Test Plan
bench/tools/bulk_produce.c.bench/run_all.shwith latency-under-load enabled to verify sample capture, overlap verification, and summary output.