Skip to content

Record timing per dispatch, not per operation - #69

Merged
cymbalrush merged 1 commit into
apple:mainfrom
cymbalrush:fix/delegate-benchmarking
Aug 26, 2026
Merged

Record timing per dispatch, not per operation#69
cymbalrush merged 1 commit into
apple:mainfrom
cymbalrush:fix/delegate-benchmarking

Conversation

@cymbalrush

Copy link
Copy Markdown
Contributor

The runtime measures a compile identifier -- a possibly-fused group of Core AI operations -- as a whole, so a duration belongs to the group rather than to any one member. It also reports each encoder twice; only one of the two measurements is kept now, since pooling them put two different quantities in one sample list.

  • OperationTiming carries op_ids and one undivided measurement, and the summary lists every member rather than a representative, which collided.
  • _timings is keyed on (odix_id, sorted op ids): delegate_id is a per-dispatch counter, and an unsorted key split one dispatch across rows.
  • An untimed warmup iteration, since a first inference is not the steady state.
  • operations_by_id exposes every operation in the function, so a caller can compute coverage on its own terms.
  • Module timings are filed against the deepest module containing all of a dispatch's operations, and no module reports a total: fusion crosses module boundaries, so a total charges one module for a sibling's work.
  • Removed total_duration_ns and the per-operation lookups, which presented a profiling span and a shared duration as latency and per-operation cost.

Adds timing_diff for comparing two runs. Op ids are positional, so an edit renumbers them and comparing by id compares unrelated work. op_id_alignment supplies the correspondence, and dispatches pair only when they cover the same operations; one that gained or lost operations is reported as resized, with the delta attributable to them, and anything else as present on one side only.

The runtime measures a compile identifier -- a possibly-fused group of Core AI
operations -- as a whole, so a duration belongs to the group rather than to any
one member. It also reports each encoder twice; only one of the two measurements
is kept now, since pooling them put two different quantities in one sample list.

- `OperationTiming` carries `op_ids` and one undivided `measurement`, and the
  summary lists every member rather than a representative, which collided.
- `_timings` is keyed on `(odix_id, sorted op ids)`: `delegate_id` is a
  per-dispatch counter, and an unsorted key split one dispatch across rows.
- An untimed warmup iteration, since a first inference is not the steady state.
- `operations_by_id` exposes every operation in the function, so a caller can
  compute coverage on its own terms.
- Module timings are filed against the deepest module containing all of a
  dispatch's operations, and no module reports a total: fusion crosses module
  boundaries, so a total charges one module for a sibling's work.
- Removed `total_duration_ns` and the per-operation lookups, which presented a
  profiling span and a shared duration as latency and per-operation cost.

Adds `timing_diff` for comparing two runs. Op ids are positional, so an edit
renumbers them and comparing by id compares unrelated work. `op_id_alignment`
supplies the correspondence, and dispatches pair only when they cover the same
operations; one that gained or lost operations is reported as resized, with the
delta attributable to them, and anything else as present on one side only.

Rendering moves onto the shared table and tree writers.
@cymbalrush
cymbalrush force-pushed the fix/delegate-benchmarking branch from 7620faf to b226718 Compare August 26, 2026 17:59
@cymbalrush
cymbalrush merged commit 431652a into apple:main Aug 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants