Skip to content

Add physical observations - #1050

Open
madsbk wants to merge 3 commits into
rapidsai:mainfrom
madsbk:physical-observations
Open

Add physical observations#1050
madsbk wants to merge 3 commits into
rapidsai:mainfrom
madsbk:physical-observations

Conversation

@madsbk

@madsbk madsbk commented Aug 28, 2026

Copy link
Copy Markdown
Member

This PR introduces physical observations. One covers a single transfer occupying a thread or a connection: one thread-pool task locally, one HTTP range request remotely, one per attempt when a request is retried. It starts when a worker picks the task up or when the request goes on the wire, and ends when that chunk is done, so its span is the transfer rather than the call that was waiting for a thread.

Summary is unchanged. No new field, no new accessor. SummaryMonitor takes an ObservationKind and computes the same statistics over whichever kind of observation you asked for.

logical = kvikio.SummaryMonitor(kvikio.ObservationKind.LOGICAL)
physical = kvikio.SummaryMonitor(kvikio.ObservationKind.PHYSICAL)

What is covered

Every byte of a call appears exactly once across its transfers. A monitor subscribed to one kind sees the same totals as a monitor subscribed to the other, so the two are comparable and a physical-only summary is never short.

That is what forces a record on the single-shot paths. FileHandle::read()/write(), MmapHandle::read() and RemoteHandle::read() do the work inline with no queue in front, so a physical record there duplicates the logical one exactly. It is emitted regardless.

path one observation is
local POSIX and GDS one thread-pool task
mmap one thread-pool task
remote MULTI_POLL one sub-range request, one per attempt
remote EASY_THREADPOOL one thread-pool task

The bounce-buffer loop inside posix_io.cpp and the per-chunk device copies are deliberately not subdivided.

Overhead

monitor_count becomes one count per kind, so a program with only a SummaryMonitor never builds a physical record. It pays 3.4 ns per task to find that out.

With a monitor subscribed, an observation costs about 60 ns whichever kind it is, since it is the same record through the same registry. On my local workstation, the four states interleaved round by round:

workload off (ns/call) logical physical both
mmap read, 8 B, 1 task 378 +62 +60 +124
host pread, 4 KiB, 1 task 3952 +211 +60 +277
host pread, 16 MiB, 16 tasks 1659380 noise noise noise

The last row is the one where the kinds differ in volume, sixteen physical records against one logical. Neither state separates from off: roughly 1 µs of observation on a 1.66 ms call, against a run-to-run spread of ±10 µs.

Notable

Observation gains parent_id, a std::optional<std::uint64_t> holding the id of the logical operation a transfer belongs to. It is empty only when the call started with no monitor registered for logical observations, since there is no logical record to point at. Nothing else leaves it empty, so a consumer subscribed to both kinds can rely on it.

A retried request is one observation per attempt, so a backoff is a gap between two records rather than one long transfer. A failed attempt is recorded with zero bytes.

Within a request there is no progress reporting, so an unsplit GET appears at its completion however long it ran. pread() splits at task_size on both remote backends, 4 MB by default. RemoteHandle::read() does not split, so there a request is as large as the call.

Nothing subscribes to PHYSICAL by default, and SummaryMonitor still defaults to LOGICAL.

Follow-ups

A trace monitor on the physical observations, writing one record per transfer for a timeline viewer. That is what makes parent_id visible to a user, and it covers what #967 asks for without a logger call on the I/O path.

Per-call queue wait, #1044, which is a consumer subscribed to both kinds and grouping by parent_id.

@madsbk madsbk self-assigned this Aug 28, 2026
@madsbk madsbk added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Aug 28, 2026
@madsbk
madsbk force-pushed the physical-observations branch from 6868555 to 81f5807 Compare August 28, 2026 13:11
@madsbk
madsbk force-pushed the physical-observations branch from 81f5807 to 763297e Compare August 28, 2026 14:05
@madsbk
madsbk marked this pull request as ready for review August 28, 2026 14:26
@madsbk
madsbk requested review from a team as code owners August 28, 2026 14:26
@rapidsai rapidsai deleted a comment from copy-pr-bot Bot Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Improves an existing functionality non-breaking Introduces a non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant