Skip to content

Add OnnxDiscrepancyCheck speedup metric with default timing updates#2502

Open
xadupre wants to merge 6 commits into
mainfrom
xadupre/speedup
Open

Add OnnxDiscrepancyCheck speedup metric with default timing updates#2502
xadupre wants to merge 6 commits into
mainfrom
xadupre/speedup

Conversation

@xadupre

@xadupre xadupre commented Jun 5, 2026

Copy link
Copy Markdown
Member

Describe your changes

Added speedup measurement for OnnxDiscrepancyCheck and updated behavior based on review feedback:

  • Changed timing_iterations default from 10 to 5.
  • If timing_iterations is set to 0, speedup measurement is skipped.
  • Added unit tests to validate the new default and the skip behavior.

Checklist before requesting a review

  • Add unit tests for this change.
  • Make sure all tests can pass.
  • Update documents if necessary.
  • Lint and apply fixes to your code by running lintrunner -a
  • Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

(Optional) Issue link

Comment thread olive/passes/onnx/discrepancy_check.py
Copilot AI changed the title add speedup to OnnxDiscrepancyCheck Add OnnxDiscrepancyCheck speedup metric with default timing updates Jun 5, 2026
@xadupre xadupre marked this pull request as ready for review June 5, 2026 17:28
Copilot AI review requested due to automatic review settings June 5, 2026 17:28
Comment thread test/passes/onnx/test_discrepancy_check.py Fixed
Comment thread test/passes/onnx/test_discrepancy_check.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances the OnnxDiscrepancyCheck pass by adding an inference speedup measurement (ONNX vs PyTorch) and introducing configurable warmup/timing iteration settings, with updated defaults and tests to validate the new behavior.

Changes:

  • Added warmup_iterations and timing_iterations config parameters (defaulting timing_iterations to 5) and implemented speedup measurement with an option to skip when timing_iterations=0.
  • Updated session/device setup to target the configured accelerator (with CPU fallback) and run PyTorch on the matched torch device.
  • Added unit tests to validate the new default and the skip behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
olive/passes/onnx/discrepancy_check.py Adds speedup timing, new config params/defaults, and device-aware session/model handling.
test/passes/onnx/test_discrepancy_check.py Adds tests for the new timing_iterations default and the “skip when 0” behavior.

Comment thread olive/passes/onnx/discrepancy_check.py
Comment on lines +369 to 371
return speedup

def compare_generation(self, config: type[BasePassConfig], ref_model) -> int:
Copilot AI requested a review from xiaoyu-work June 5, 2026 20:47
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants