[ml] Fix jobs partial update across all job types#47819
Open
Chakradhar886 wants to merge 6 commits into
Open
Conversation
Root bug: ml_client.jobs.get(name) -> mutate -> jobs.create_or_update(job) fails for AutoML, Command, Pipeline, Sweep and Spark. Each job type raises a different serializer or MFE PUT comparator error (QueueSettings._to_rest_object AttributeError for Sweep, InputBindings/output-binding UserError for Pipeline and Spark, registry-scope UserError for Command, 'Input path can't be empty' for AutoML). This change adds two intentionally narrow fixes: 1. A new public jobs.update(name, *, display_name, description, tags, properties) API that routes through the RunHistory PATCH endpoint used by ML Studio's portal for inline edits. The endpoint is job-type agnostic and merges tags/properties instead of replacing them. 2. A temporary shortcut inside create_or_update that detects the 'fetched job with only metadata changes' case (job carries an ARM id, no compute/experiment_name kwargs supplied) and routes it through the same RunHistory PATCH path. Any failure inside the shortcut falls through to the original create/update logic so brand-new job creation is bit-for-bit unchanged. Verified live against AutoML, Command, Pipeline, Sweep and Spark jobs; MFE GET (api-version 2024-10-01) confirms displayName and merged tags persisted server-side.
Chakradhar886
requested review from
JustinFirsching,
NonStatic2014,
achauhan-scc,
arunsu,
jayesh-tanna,
kingernupur and
nick863
as code owners
Chakradhar886
requested review from
novaturient95,
rtanase,
sharma-riti and
vivram
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses a bug in azure-ai-ml where the common "fetch a job, mutate it, resubmit via jobs.create_or_update(job)" pattern fails for AutoML, Command, Pipeline, Sweep, and Spark jobs (each raising a different serializer or MFE PUT comparator error). The change touches a single file, sdk/ml/azure-ai-ml/azure/ai/ml/operations/_job_operations.py, and introduces two related fixes that route metadata-only edits through the RunHistory PATCH endpoint used by ML Studio's portal.
Changes:
- Adds a new public
JobOperations.update(name, *, display_name, description, tags, properties)API that patches job metadata uniformly across all job types via RunHistoryadd_or_modify_by_experiment_name. - Adds a "TEMPORARY HOTFIX" shortcut inside
create_or_updatethat detects a fetched job (ARMidpresent, nocompute/experiment_namekeyword override) and routes it through the same RunHistory PATCH path, falling back to the original create/update flow on any exception. - Reuses the existing pipeline-handling pattern (
_get_job→_get_job_2401refetch →PipelineChildJobErrorrejection) already used byarchive/restore.
| refreshed = self._get_job(name) | ||
| if refreshed.properties.job_type == RestJobType.PIPELINE: | ||
| refreshed = self._get_job_2401(name) | ||
| return Job._from_rest_object(refreshed) |
Comment on lines
+958
to
+966
| def update( | ||
| self, | ||
| name: str, | ||
| *, | ||
| display_name: Optional[str] = None, | ||
| description: Optional[str] = None, | ||
| tags: Optional[Dict[str, str]] = None, | ||
| properties: Optional[Dict[str, str]] = None, | ||
| ) -> Job: |
Comment on lines
+958
to
+966
| def update( | ||
| self, | ||
| name: str, | ||
| *, | ||
| display_name: Optional[str] = None, | ||
| description: Optional[str] = None, | ||
| tags: Optional[Dict[str, str]] = None, | ||
| properties: Optional[Dict[str, str]] = None, | ||
| ) -> Job: |
Comment on lines
+742
to
+747
| if ( | ||
| getattr(job, "id", None) | ||
| and getattr(job, "name", None) | ||
| and compute is None | ||
| and experiment_name is None | ||
| ): |
| if _is_pipeline_child_job(job_object): | ||
| raise PipelineChildJobError(job_id=job_object.id) | ||
|
|
||
| from azure.ai.ml._restclient.runhistory.models import CreateRun |
saanikaguptamicrosoft
left a comment
saanikaguptamicrosoft
left a comment
Member
There was a problem hiding this comment.
Pls add an entry in CHANGELOG
…a RunHistory; add CHANGELOG + api.md - Narrow the try/except in the create_or_update shortcut to only guard the RunHistory PATCH; post-PATCH refresh/resolve failures now surface directly instead of silently falling back to the buggy MFE PUT path (addresses Copilot / saanikaguptamicrosoft review). - Route _archive_or_restore through the same RunHistory shortcut so jobs.archive/restore work for Command/Sweep/Spark/Parallel (were previously broken for all non-Pipeline types due to the same MFE PUT round-trip bug). - Add CHANGELOG entry for jobs.update() and the create_or_update fix. - Add jobs.update signature to api.md.
…-metadata-shortcut
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root bug: ml_client.jobs.get(name) -> mutate -> jobs.create_or_update(job) fails for AutoML, Command, Pipeline, Sweep and Spark. Each job type raises a different serializer or MFE PUT comparator error (QueueSettings._to_rest_object AttributeError for Sweep, InputBindings/output-binding UserError for Pipeline and Spark, registry-scope UserError for Command, 'Input path can't be empty' for AutoML).
This change adds two intentionally narrow fixes:
A new public jobs.update(name, *, display_name, description, tags, properties) API that routes through the RunHistory PATCH endpoint used by ML Studio's portal for inline edits. The endpoint is job-type agnostic and merges tags/properties instead of replacing them.
A temporary shortcut inside create_or_update that detects the 'fetched job with only metadata changes' case (job carries an ARM id, no compute/experiment_name kwargs supplied) and routes it through the same RunHistory PATCH path. Any failure inside the shortcut falls through to the original create/update logic so brand-new job creation is bit-for-bit unchanged.
Verified live against AutoML, Command, Pipeline, Sweep and Spark jobs; MFE GET (api-version 2024-10-01) confirms displayName and merged tags persisted server-side.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines