Add I/O statistics: totals of what a run read and wrote

[madsbk]

Builds on the observation facility from #1033. That PR gives a callback per user-facing I/O call, and this one gives the answer most people actually want from it, which is what a run did in total.

SummaryMonitor registers itself on construction and accumulates while it exists. Summary carries the operations, the bytes, the errors, and the time at least one operation was in flight, which is what distinguishes an I/O-bound run from a compute-bound one.

Using it

with kvikio.SummaryMonitor() as monitor:
    ...
print(monitor.get())

In C++ the constructor takes an optional callback, which runs on destruction, so a program can report its own I/O without touching the code that performs it.

kvikio::statistics::SummaryMonitor const monitor{
  [](kvikio::statistics::Summary const& summary) { std::cout << summary.report(); }};

An interval is the difference of two readings, so periodic reporting takes one reading per tick and differences it against the last. Taking the reading and the interval separately would leave a gap that an operation could fall into twice.

baseline = monitor.get()
while running:
    time.sleep(interval)
    now = monitor.get()
    report(now.since(baseline))
    baseline = now

The report

Printing a summary, or report(), gives a report meant for a person, always the same shape so two runs can be compared line by line. This is python/kvikio/examples/hello_world.py:

KvikIO I/O summary
  wall time            251.72 ms
  busy time            5.95 ms (2.36 % of the wall time)
  busy bandwidth       538.15 kB/s
  operations           5 (4 read, 1 write)
  time per operation   1.19 ms mean, 3.19 ms longest
  bytes                3.12 KiB of 3.12 KiB requested (2.34 KiB read, 800 B written)
  errors               0
  backend POSIX        3.12 KiB in 5 ops, 5.95 ms, 538.15 kB/s
  backend GDS          unused
  backend MMAP         unused
  backend REMOTE_HTTP  unused
  backend REMOTE_HDFS  unused

Five operations for one write and four reads, however many reads KvikIO issued underneath, since the observations are logical.

Busy time is the union of the operations' spans, so overlapping work counts once and the gaps between calls count as idle. Busy bandwidth divides by that rather than by the wall time, so a program that reads for 10 ms and then computes for 90 ms is not reported as ten times slower than its storage really is. The backend rows are the only place the report says whether a read reached cuFile or fell back to POSIX, which compatibility mode decides per call.

Overhead

On my local workstation, 32 cores, medians over two million observed operations, so an order of magnitude rather than a specification.

per operation
no monitor registered	4.8 ns
SummaryMonitor	64 to 79 ns

This includes everything, both the observation facility from #1033 stamping and dispatching each operation, and the accumulation this PR adds on top. So a monitor costs about 2 % of a 4 KiB pread() and 0.25 % of a 1 MiB one. Registering none is the 4.8 ns row, one relaxed atomic load per operation.

Eight threads doing nothing but emitting observations cost 320 ns each, every operation taking the monitor's lock to add itself to the totals. Real work in between makes it disappear.

Follow-ups

• What KvikIO spends on itself, the bounce buffers it allocates, the connections it opens, the time inside the file system. The next PR adds those counters and surfaces them on Summary.
• A record per operation, which a summary cannot give: a timeline monitor and a sampling monitor.
• ObservationKind::PHYSICAL. Everything here is logical, one user-facing call being one operation. When physical operations arrive, Summary stays one type rather than splitting in two, since the fields mean the same thing at either level.

[madsbk]

We can now use the new Monitor instead of this homemade Timer

[madsbk]

This file and its .cpp is mostly a copy-paste from https://github.com/rapidsai/rapidsmpf/blob/main/cpp/include/rapidsmpf/utils/string.hpp#L73

[kingcrimsontianyu]

Great framework. This opens doors to many statistics we want to collect such as p50/p90/ p99 latency and probably TTFB (time to first byte).

[madsbk]

/merge