Add an I/O observation facility by madsbk · Pull Request #1033 · rapidsai/kvikio

madsbk · 2026-08-14T08:51:32Z

This PR introduces a hook into monitoring KvikIO operations, with the goal of building statistics, Quent timelines, and whatever else wants to know what the I/O layer is doing. This PR is the base. A follow-up introduces a concrete Monitor that makes statistics easy to get.

The hook reports whole KvikIO operations (the logical level) so a pread() is a single observation however many reads the thread pool issued underneath. Physical operations can be added along the same path later, which could be the basis of #1016.

Each call produces a kvikio::Observation: its span, the offset and size etc. To receive them, derive from kvikio::Monitor and register it. A monitor is told when an operation starts as well as when it finishes.

// Example of a Monitor that tracks how many KvikIO operations are in flight at any moment.
class QueueDepth : public kvikio::Monitor {
  void on_start(kvikio::Observation const&) noexcept override { ++_in_flight; }
  void on_finish(kvikio::Observation const&) noexcept override { --_in_flight; }
  std::atomic<int> _in_flight{0};
};

QueueDepth gauge;
auto id = kvikio::register_monitor(&gauge);

Overhead

Measured on my local workstation:

~3 ns per call when nobody is observing, which is a gate check and a branch.
~60 ns per call when somebody is, or 1 % of a 64 KiB pread(), and nothing detectable at a megabyte.

Confirmed against a real workload: cudf-polars PDS-H query 1 at scale 10, with and without a monitor attached, showed no difference outside noise.

What is not observed

The cuFile asynchronous API on a GDS system, and the batch API, complete without KvikIO seeing it, so they emit nothing. Handling those needs a stream-completion callback, which is future work.

Follow-up: statistics

The next PR adds kvikio::statistics::SummaryMonitor, which is a Monitor and nothing more:

monitor = kvikio.SummaryMonitor()   # statistics are now on
...
print(monitor.get())

KvikIO I/O summary
  wall time            1.876 s
  busy time            366.592 ms (19.54 % of the wall time)
  busy bandwidth       8.95 GB/s
  operations           7970
  bytes requested      3.06 GiB
  bytes transferred    3.06 GiB
  errors               0

Those are real numbers, from a cudf-polars run, and they show a very useful busy bandwidth. 8.95 GB/s is the rate while KvikIO actually had work in flight, where dividing the same bytes by the wall clock would have said 1.75 GB/s and described the query rather than the storage.

A TimelineMonitor, for when things happened rather than how much, is planned after that.

`nbytes()` caches the file size and `write()` invalidates it, but two `pwrite()` paths never go through `write()`: the host path and the sub-threshold device shortcut. So `nbytes()` could report a stale size after a write. Both now invalidate the cache once the write has completed, so a `nbytes()` call racing with an in-flight `pwrite()` cannot leave a stale size cached either. Unrelated to the rest of this branch.

madsbk · 2026-08-14T11:55:49Z

-  if (is_read_out_of_bounds(file_offset, size, _nbytes)) {
-    std::stringstream ss;
-    ss << "cannot read " << file_offset << "+" << size << " bytes into a " << _nbytes
-       << " bytes file (" << _endpoint->str() << ")";
-    KVIKIO_FAIL(ss.str(), std::invalid_argument);
-  }


Not removed, just moved to the top of read, before the buffer is classified.

madsbk · 2026-08-14T11:59:14Z

-  if (is_read_out_of_bounds(file_offset, size, _nbytes)) {
-    std::stringstream ss;
-    ss << "cannot read " << file_offset << "+" << size << " bytes into a " << _nbytes
-       << " bytes file (" << _endpoint->str() << ")";
-    KVIKIO_FAIL(ss.str(), std::invalid_argument);
-  }


Not removed, just moved to the top of read, before the buffer is classified.

wence-

Broadly looks good, I think.

The number of `IoBackend` values, for a table with one entry per backend. Derived from the last enumerator and placed next to the enum, so the two stay together.

…l-observations

`Clock` is monotonic, so its timestamps cannot be compared with anything outside the process. `ClockAnchor::now()` reads it together with the wall clock, and `to_wall_clock()` maps a timestamp through that pair. This keeps the measurements on a clock that cannot step while still allowing an observation to be lined up with a log line, another process, or a profiler trace.

Both repeated what the documentation of the function beside them already says.

vyasr

Approving CMake (didn't review the C++).

madsbk · 2026-08-18T05:23:20Z

But register_monitor is the public API

…

On Tue, 18 Aug 2026 at 07.15, Tianyu Liu ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In cpp/src/observation.cpp <#1033 (comment)>: > +std::string_view to_string(ObservationKind kind) noexcept +{ + switch (kind) { + case ObservationKind::LOGICAL: return "LOGICAL"; + default: return "Unknown"; + } +} + +std::uint64_t register_monitor(Monitor* monitor, ObservationKind kind) +{ + return Registry::instance().add(monitor, kind); +} + +void unregister_monitor(std::uint64_t id) { Registry::instance().remove(id); } + +namespace detail { A better location for this section is src/detail/observation_recorder.cpp. This would improve consistency. — Reply to this email directly, view it on GitHub <#1033?email_source=notifications&email_token=AAH6FQB4RLXW762UEFPP3D35KPQ7RA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTIOJVG42DMOBQHA42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#pullrequestreview-4957468089>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH6FQG5DE33JUI2L2V5QFT5KPQ7RAVCNFSNUABFKJSXA33TNF2G64TZHM2DKMZRGQ3TMMRZHNEXG43VMU5TKMJUHE4TQMJRG44KC5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AAH6FQBW3UBE7F42ABAFDL35KPQ7RA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTIOJVG42DMOBQHA42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/AAH6FQBWGJNGK7P7PW26P3T5KPQ7RA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTIOJVG42DMOBQHA42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

kingcrimsontianyu · 2026-08-18T05:26:04Z

Oh I was commenting on the namespace detail part.

madsbk · 2026-08-18T05:27:28Z

But do you really want to split that into two files ?

…

On Tue, 18 Aug 2026 at 07.26, Tianyu Liu ***@***.***> wrote: *kingcrimsontianyu* left a comment (rapidsai/kvikio#1033) <#1033 (comment)> Oh I was commenting on the namespace detail part. — Reply to this email directly, view it on GitHub <#1033?email_source=notifications&email_token=AAH6FQGTJYYIO6J2GMRHMZT5KPSIHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZSGQYDAMJZGMZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5324001933>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH6FQE6T53JEW7UC7JIO7T5KPSIHAVCNFSNUABFKJSXA33TNF2G64TZHM2DKMZRGQ3TMMRZHNEXG43VMU5TKMJUHE4TQMJRG44KC5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AAH6FQFDNZS4OYFI52QIYI35KPSIHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZSGQYDAMJZGMZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/AAH6FQGNK37AOZOPI65JLXD5KPSIHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZSGQYDAMJZGMZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

kingcrimsontianyu · 2026-08-18T05:32:04Z

It's up to you. We do have an existing src/detail directory containing code that otherwise goes to the detail namespace. Not a big deal though.

madsbk · 2026-08-18T05:36:47Z

Hmm alright, let me think about this. It might be nice to have a clean header that only have the public APl. I can also move the small impl to .cpp.

…

On Tue, 18 Aug 2026 at 07.32, Tianyu Liu ***@***.***> wrote: *kingcrimsontianyu* left a comment (rapidsai/kvikio#1033) <#1033 (comment)> It's up to you. We do have an existing src/detail directory containing code that otherwise goes to the detail namespace. Not a big deal though. — Reply to this email directly, view it on GitHub <#1033?email_source=notifications&email_token=AAH6FQEMVIH6PXDRK234YFD5KPS6TA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZSGQYDINBRGAY2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5324044101>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH6FQBT7OICNR3QNPBRVHD5KPS6TAVCNFSNUABFKJSXA33TNF2G64TZHM2DKMZRGQ3TMMRZHNEXG43VMU5TKMJUHE4TQMJRG44KC5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AAH6FQBS6L5QXCLGFHQXGP35KPS6TA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZSGQYDINBRGAY2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/AAH6FQHDKS4CF3VE5LW3FQ35KPS6TA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZSGQYDINBRGAY2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

kingcrimsontianyu

Thanks. Good design!

…re a write

madsbk · 2026-08-18T07:54:01Z

/merge

Builds on the observation facility from #1033. That PR gives a callback per user-facing I/O call, and this one gives the answer most people actually want from it, which is what a run did in total. `SummaryMonitor` registers itself on construction and accumulates while it exists. `Summary` carries the operations, the bytes, the errors, and the time at least one operation was in flight, which is what distinguishes an I/O-bound run from a compute-bound one. ### Using it ```python with kvikio.SummaryMonitor() as monitor: ... print(monitor.get()) ``` In C++ the constructor takes an optional callback, which runs on destruction, so a program can report its own I/O without touching the code that performs it. ```c++ kvikio::statistics::SummaryMonitor const monitor{ [](kvikio::statistics::Summary const& summary) { std::cout << summary.report(); }}; ``` An interval is the difference of two readings, so periodic reporting takes one reading per tick and differences it against the last. Taking the reading and the interval separately would leave a gap that an operation could fall into twice. ```python baseline = monitor.get() while running: time.sleep(interval) now = monitor.get() report(now.since(baseline)) baseline = now ``` ### The report Printing a summary, or `report()`, gives a report meant for a person, always the same shape so two runs can be compared line by line. This is `python/kvikio/examples/hello_world.py`: ``` KvikIO I/O summary wall time 251.72 ms busy time 5.95 ms (2.36 % of the wall time) busy bandwidth 538.15 kB/s operations 5 (4 read, 1 write) time per operation 1.19 ms mean, 3.19 ms longest bytes 3.12 KiB of 3.12 KiB requested (2.34 KiB read, 800 B written) errors 0 backend POSIX 3.12 KiB in 5 ops, 5.95 ms, 538.15 kB/s backend GDS unused backend MMAP unused backend REMOTE_HTTP unused backend REMOTE_HDFS unused ``` Five operations for one write and four reads, however many reads KvikIO issued underneath, since the observations are logical. **Busy time** is the union of the operations' spans, so overlapping work counts once and the gaps between calls count as idle. **Busy bandwidth** divides by that rather than by the wall time, so a program that reads for 10 ms and then computes for 90 ms is not reported as ten times slower than its storage really is. The **backend** rows are the only place the report says whether a read reached cuFile or fell back to POSIX, which compatibility mode decides per call. ### Overhead On my local workstation, 32 cores, medians over two million observed operations, so an order of magnitude rather than a specification. | per operation | | |---|---| | no monitor registered | 4.8 ns | | `SummaryMonitor` | 64 to 79 ns | This includes everything, both the observation facility from #1033 stamping and dispatching each operation, and the accumulation this PR adds on top. So a monitor costs about 2 % of a 4 KiB `pread()` and 0.25 % of a 1 MiB one. Registering none is the 4.8 ns row, one relaxed atomic load per operation. Eight threads doing nothing but emitting observations cost 320 ns each, every operation taking the monitor's lock to add itself to the totals. Real work in between makes it disappear. ### Follow-ups - **What KvikIO spends on itself**, the bounce buffers it allocates, the connections it opens, the time inside the file system. The next PR adds those counters and surfaces them on `Summary`. - **A record per operation**, which a summary cannot give: a timeline monitor and a sampling monitor. - **`ObservationKind::PHYSICAL`**. Everything here is logical, one user-facing call being one operation. When physical operations arrive, `Summary` stays one type rather than splitting in two, since the fields mean the same thing at either level. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Bradley Dice (https://github.com/bdice) - Tianyu Liu (https://github.com/kingcrimsontianyu) URL: #1036

madsbk self-assigned this Aug 14, 2026

madsbk added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Aug 14, 2026

madsbk force-pushed the logical-observations branch 3 times, most recently from d42da15 to 0dece5b Compare August 14, 2026 10:50

madsbk force-pushed the logical-observations branch 2 times, most recently from e1bf9bf to 7360ca8 Compare August 14, 2026 11:52

madsbk commented Aug 14, 2026

View reviewed changes

madsbk force-pushed the logical-observations branch from 7360ca8 to 0846f46 Compare August 14, 2026 12:10

Add an I/O observation facility

9a4fe49

madsbk force-pushed the logical-observations branch from 0846f46 to 9a4fe49 Compare August 14, 2026 12:12

madsbk marked this pull request as ready for review August 14, 2026 12:55

madsbk requested review from a team as code owners August 14, 2026 12:55

rapidsai deleted a comment from copy-pr-bot Bot Aug 14, 2026

madsbk requested a review from kingcrimsontianyu August 14, 2026 12:56

wence- reviewed Aug 14, 2026

View reviewed changes

Comment thread cpp/include/kvikio/observation.hpp Outdated

Comment thread cpp/src/file_handle.cpp

Comment thread cpp/src/observation.cpp Outdated

Comment thread cpp/src/remote_handle.cpp Outdated

Comment thread cpp/tests/test_observation.cpp Outdated

based on reviews

11f3c68

madsbk requested a review from wence- August 14, 2026 14:20

kingcrimsontianyu reviewed Aug 14, 2026

View reviewed changes

Comment thread cpp/include/kvikio/observation.hpp

Use ALL_CAPS enumerators

36bc839

madsbk requested a review from kingcrimsontianyu August 14, 2026 14:39

madsbk added 2 commits August 14, 2026 22:47

Add num_io_backends

cb681af

The number of `IoBackend` values, for a table with one entry per backend. Derived from the last enumerator and placed next to the enum, so the two stay together.

Merge branch 'main' of https://github.com/rapidsai/kvikio into logica…

a37a5aa

…l-observations

madsbk force-pushed the logical-observations branch from fc7f400 to d02ce30 Compare August 15, 2026 11:39

madsbk added 2 commits August 15, 2026 13:43

Add Observation::source

e75159b

madsbk added 2 commits August 15, 2026 13:43

Make the cached file size atomic

9b40af8

cleanup

219c84d

madsbk force-pushed the logical-observations branch from d02ce30 to 219c84d Compare August 15, 2026 11:46

Drop redundant comments

526feca

Both repeated what the documentation of the function beside them already says.

wence- approved these changes Aug 17, 2026

View reviewed changes

vyasr approved these changes Aug 17, 2026

View reviewed changes

kingcrimsontianyu reviewed Aug 18, 2026

View reviewed changes

Comment thread cpp/include/kvikio/detail/observation_recorder.hpp Outdated

kingcrimsontianyu reviewed Aug 18, 2026

View reviewed changes

Comment thread cpp/src/observation.cpp

kingcrimsontianyu reviewed Aug 18, 2026

View reviewed changes

Comment thread cpp/include/kvikio/observation.hpp

madsbk added 2 commits August 18, 2026 08:17

Say why the clock anchor's conversion is noexcept

e92f751

moving impl to cpp/src/detail/observation_recorder.cpp

9c7e0dd

kingcrimsontianyu approved these changes Aug 18, 2026

View reviewed changes

Reject a call before observing it, and invalidate the size cache befo…

c223337

…re a write

rapids-bot Bot merged commit b7b967e into rapidsai:main Aug 18, 2026
65 checks passed

madsbk deleted the logical-observations branch August 18, 2026 08:03

madsbk mentioned this pull request Aug 18, 2026

Add I/O statistics: totals of what a run read and wrote #1036

Merged

Matt711 mentioned this pull request Aug 25, 2026

[Story]: Instrument I/O concurrency and partition-sizing signals across cudf-polars, rapidsmpf, kvikio NVIDIA/cudf#23810

Open

Uh oh!

Conversation

madsbk commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overhead

What is not observed

Follow-up: statistics

Uh oh!

madsbk Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

madsbk Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

wence- left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vyasr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

madsbk commented Aug 18, 2026 via email

Uh oh!

kingcrimsontianyu commented Aug 18, 2026

Uh oh!

madsbk commented Aug 18, 2026 via email

Uh oh!

kingcrimsontianyu commented Aug 18, 2026

Uh oh!

madsbk commented Aug 18, 2026 via email

Uh oh!

Uh oh!

kingcrimsontianyu left a comment

Choose a reason for hiding this comment

Uh oh!

madsbk commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

madsbk commented Aug 14, 2026 •

edited

Loading

madsbk Aug 14, 2026 •

edited

Loading