dumps OnnxDiscrepancyCheck result on disc so they can be picked up later and gathered for a dashboard

[xadupre]

Describe your changes

The pass OnnxDiscrepancyCheck is used to gather metrics on canary models. They need to be dumped to let the user see them in detail, the user can be a dashboard as well.

When --test is active, add_discrepancy_check_pass now injects max_mae: 0.1 and report_output_dir (set to the run config's output_dir) into the discrepancy check pass configuration alongside reference_model_path.

Fixed test regressions in test_workflow_run_command_with_test_override and test_workflow_run_command_with_test_reuses_test_output_dir:

• Set mock_run.return_value = None so save_discrepancy_check_results exits early at its null-guard check instead of attempting to serialize a MagicMock via json.dumps.
• Updated the assert_called_once_with expectation to include max_mae and report_output_dir in the expected discrepancy check pass config.

Checklist before requesting a review

• Add unit tests for this change.
• Make sure all tests can pass.
• Update documents if necessary.
• Lint and apply fixes to your code by running lintrunner -a
• Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

(Optional) Issue link

[xadupre]

@copilot test\cli\test_cli.py:173: in test_workflow_run_command_with_test_override
cli_main(command_args)
olive\cli\launcher.py:79: in main
service.run()
olive\telemetry\telemetry_extensions.py:137: in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
olive\cli\run.py:96: in run
save_discrepancy_check_results(workflow_output, output_path)
olive\cli\base.py:110: in save_discrepancy_check_results
report_path.write_text(json.dumps(results, indent=2))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\ToolCache\Python\3.12.10\x64\Lib\json_init_.py:238: in dumps
**kw).encode(obj)
^^^^^^^^^^^
C:\ToolCache\Python\3.12.10\x64\Lib\json\encoder.py:202: in encode
chunks = list(chunks)
^^^^^^^^^^^^
C:\ToolCache\Python\3.12.10\x64\Lib\json\encoder.py:439: in _iterencode
o = _default(o)
^^^^^^^^^^^
C:\ToolCache\Python\3.12.10\x64\Lib\json\encoder.py:180: in default
raise TypeError(f'Object of type {o.class.name} '
E TypeError: Object of type MagicMock is not JSON serializable

[Copilot]

@copilot test\cli\test_cli.py:173: in test_workflow_run_command_with_test_override
cli_main(command_args)
olive\cli\launcher.py:79: in main
...

Fixed in the latest commit. The test was patching olive.workflows.run but the default MagicMock return value was truthy, so save_discrepancy_check_results didn't exit early and tried to serialize the mock via json.dumps. Added mock_run.return_value = None so the function returns immediately at the if not workflow_output guard.

[xadupre]

@copilot test/cli/test_cli.py:177: in test_workflow_run_command_with_test_override
mock_run.assert_called_once_with(
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/unittest/mock.py:961: in assert_called_once_with
return self.assert_called_with(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/unittest/mock.py:949: in assert_called_with
raise AssertionError(_error_message()) from cause
E AssertionError: expected call not found.
E Expected: run({'input_model': {'type': 'HfModel', 'model_path': 'hf-internal-testing/tiny-random-LlamaForCausalLM', 'load_kwargs': {'attn_implementation': 'eager', 'trust_remote_code': False}, 'test_model_config': {'hidden_layers': 2}, 'test_model_path': '/mnt/vss/_work/1/.pytest_basetemp/test_workflow_run_command_with1/output/test_model'}, 'output_dir': '/mnt/vss/_work/1/.pytest_basetemp/test_workflow_run_command_with1/output', 'passes': {'discrepancy_check': {'type': 'OnnxDiscrepancyCheck', 'reference_model_path': '/mnt/vss/_work/1/.pytest_basetemp/test_workflow_run_command_with1/output/test_model'}}}, list_required_packages=False, package_config=None, tempdir=None)
E Actual: run({'input_model': {'type': 'HfModel', 'model_path': 'hf-internal-testing/tiny-random-LlamaForCausalLM', 'load_kwargs': {'attn_implementation': 'eager', 'trust_remote_code': False}, 'test_model_config': {'hidden_layers': 2}, 'test_model_path': '/mnt/vss/_work/1/.pytest_basetemp/test_workflow_run_command_with1/output/test_model'}, 'output_dir': '/mnt/vss/_work/1/.pytest_basetemp/test_workflow_run_command_with1/output', 'passes': {'discrepancy_check': {'type': 'OnnxDiscrepancyCheck', 'reference_model_path': '/mnt/vss/_work/1/.pytest_basetemp/test_workflow_run_command_with1/output/test_model', 'max_mae': 0.1, 'report_output_dir': '/mnt/vss/_work/1/.pytest_basetemp/test_workflow_run_command_with1/output'}}}, list_required_packages=False, tempdir=None, package_config=None)

[Copilot]

@copilot test/cli/test_cli.py:177: in test_workflow_run_command_with_test_override
mock_run.assert_called_once_with(
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/unittest/mock.py:961: ...

Fixed in the latest commit. The test assertion was missing max_mae: 0.1 and report_output_dir that add_discrepancy_check_pass now injects into the pass config.

[xadupre]

@copilot test/cli/test_cli.py:244: in test_workflow_run_command_with_test_reuses_test_output_dir
cli_main(["run", "--run-config", str(config_path), "--test"])
olive/cli/launcher.py:79: in main
service.run()
olive/telemetry/telemetry_extensions.py:137: in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
olive/cli/run.py:96: in run
save_discrepancy_check_results(workflow_output, output_path)
olive/cli/base.py:110: in save_discrepancy_check_results
report_path.write_text(json.dumps(results, indent=2))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/json/init.py:238: in dumps
**kw).encode(obj)
^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/json/encoder.py:202: in encode
chunks = list(chunks)
^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/json/encoder.py:439: in _iterencode
o = _default(o)
^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/json/encoder.py:180: in default
raise TypeError(f'Object of type {o.class.name} '
E TypeError: Object of type MagicMock is not JSON serializable

[Copilot]

@copilot test/cli/test_cli.py:244: in test_workflow_run_command_with_test_reuses_test_output_dir
cli_main(["run", "--run-config", str(config_path), "--test"])
olive/cli/launcher.py:79: in main
...

Fixed in the latest commit — added mock_run.return_value = None to test_workflow_run_command_with_test_reuses_test_output_dir for the same reason as the previous test.