Add physical observations - #1050
Open
madsbk wants to merge 3 commits into
Open
Conversation
madsbk
force-pushed
the
physical-observations
branch
from
August 28, 2026 13:11
6868555 to
81f5807
Compare
madsbk
force-pushed
the
physical-observations
branch
from
August 28, 2026 14:05
81f5807 to
763297e
Compare
madsbk
marked this pull request as ready for review
August 28, 2026 14:26
…al-observations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces physical observations. One covers a single transfer occupying a thread or a connection: one thread-pool task locally, one HTTP range request remotely, one per attempt when a request is retried. It starts when a worker picks the task up or when the request goes on the wire, and ends when that chunk is done, so its span is the transfer rather than the call that was waiting for a thread.
Summaryis unchanged. No new field, no new accessor.SummaryMonitortakes anObservationKindand computes the same statistics over whichever kind of observation you asked for.What is covered
Every byte of a call appears exactly once across its transfers. A monitor subscribed to one kind sees the same totals as a monitor subscribed to the other, so the two are comparable and a physical-only summary is never short.
That is what forces a record on the single-shot paths.
FileHandle::read()/write(),MmapHandle::read()andRemoteHandle::read()do the work inline with no queue in front, so a physical record there duplicates the logical one exactly. It is emitted regardless.MULTI_POLLEASY_THREADPOOLThe bounce-buffer loop inside
posix_io.cppand the per-chunk device copies are deliberately not subdivided.Overhead
monitor_countbecomes one count per kind, so a program with only aSummaryMonitornever builds a physical record. It pays 3.4 ns per task to find that out.With a monitor subscribed, an observation costs about 60 ns whichever kind it is, since it is the same record through the same registry. On my local workstation, the four states interleaved round by round:
pread, 4 KiB, 1 taskpread, 16 MiB, 16 tasksThe last row is the one where the kinds differ in volume, sixteen physical records against one logical. Neither state separates from
off: roughly 1 µs of observation on a 1.66 ms call, against a run-to-run spread of ±10 µs.Notable
Observationgainsparent_id, astd::optional<std::uint64_t>holding theidof the logical operation a transfer belongs to. It is empty only when the call started with no monitor registered for logical observations, since there is no logical record to point at. Nothing else leaves it empty, so a consumer subscribed to both kinds can rely on it.A retried request is one observation per attempt, so a backoff is a gap between two records rather than one long transfer. A failed attempt is recorded with zero bytes.
Within a request there is no progress reporting, so an unsplit GET appears at its completion however long it ran.
pread()splits attask_sizeon both remote backends, 4 MB by default.RemoteHandle::read()does not split, so there a request is as large as the call.Nothing subscribes to
PHYSICALby default, andSummaryMonitorstill defaults toLOGICAL.Follow-ups
A trace monitor on the physical observations, writing one record per transfer for a timeline viewer. That is what makes
parent_idvisible to a user, and it covers what #967 asks for without a logger call on the I/O path.Per-call queue wait, #1044, which is a consumer subscribed to both kinds and grouping by
parent_id.