Fix resolve_job_prefix check-run scoping and arch filtering - #6
Draft
jithunnair-amd wants to merge 4 commits into
Draft
Conversation
The CUDA default/distributed/inductor shard totals were read straight from the hardcoded shard_counts in parity_job_config.json. pytorch/pytorch reshards its CUDA test matrix without notice (default 5 -> 14, distributed 3 -> 10), so every constructed "(default, i, 5)" key and "test-reports-...-5" artifact prefix missed the real "(default, i, 14)" jobs. No CUDA baseline artifacts downloaded and every parity run failed with "TEST KEY ... DOES NOT EXIST IN JOBS". Read the real shard total from the resolved run's CUDA job names (mirroring what the ROCm side already does), falling back to the config value when no matching job is found. This keeps parity in sync automatically across future upstream reshards.
Extend dynamic shard detection to every architecture (mi350, mi300, mi200, navi31, nightly, preview) and the CUDA trunk baseline, replacing the remaining hardcoded shard_counts lookups in the ROCm default/distributed/ inductor sections (primary and commit-vs-commit baseline paths). The standalone ROCm arch workflows (rocm-mi300, rocm-mi200, periodic-rocm-*, inductor-rocm-*) run their sharded test jobs inside a reusable workflow (workflow_call), so the shards do not appear in the run's jobs API - only in the commit check-runs API. derive_shard_count() now consults both sources (jobs API first, then check-runs) before falling back to the config value, so upstream reshards are picked up automatically for CUDA and ROCm alike (e.g. mi300 default 6 -> 8, inductor 2 -> 4). Validated offline against SHA 21f6559: mi300 derives default=8, distributed=3, inductor=4; mi350 default=8 (from trunk jobs API); CUDA default=14, distributed=10.
The job-name prefix per arch (e.g. linux-noble-rocm-py3.12-mi300) was hardcoded in parity_job_config.json and drifted whenever upstream bumped the OS/python/compiler or renamed a runner tag (the nightly gfx942 -> mi350 rename, py3.10 -> py3.12, etc.), each time requiring a manual config edit and silently matching zero jobs until someone noticed. Add resolve_job_prefix(): gather candidate '<prefix> / <kind> (<config>,...' names from the resolved run's jobs API and the commit check-runs API (ROCm arch shards run in a reusable workflow, so they only appear in check-runs), keep the configured prefix when it is still present, otherwise pick the candidate most similar to it, and fall back to the configured value when nothing matches. Wired into every ROCm section (default/distributed/ inductor, primary + commit-vs-commit baseline). The configured value is now just a hint that self-heals on drift. CUDA is intentionally left configured: trunk carries several deliberate CUDA families (cuda13.0 AND cuda13.2), so the baseline choice must stay explicit; each ROCm arch has exactly one family, so detection there is unambiguous. Validated offline: a stale 'gfx942' config resolves to the real 'linux-noble-rocm-py3.12-mi300', and a correct config is kept unchanged.
resolve_job_prefix previously called get_check_runs_for_commit(sha, "") and considered every check run on the commit, which could pick CUDA or other ROCm arch job prefixes when the configured prefix was stale. Scope check-runs to the resolved workflow run (including nested reusable workflow runs via referenced_workflows and details_url expansion), filter candidates to ROCm-only prefixes, and verify the detected prefix matches the requested --arch before auto-healing. Test Plan: ``` python3 -m py_compile .automation_scripts/pytorch-unit-test-scripts/download_testlogs ``` Authored with assistance from Cursor Co-authored-by: Cursor <cursoragent@cursor.com>
4 tasks
jithunnair-amd
marked this pull request as draft
July 31, 2026 23:33
Collaborator
Author
|
Disclaimer: AI-generated; I haven't reviewed this content yet |
ethanwee1
force-pushed
the
ew/parity-cuda-dynamic-shards
branch
from
August 3, 2026 20:23
340abeb to
4159f02
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix for ROCm#3523: scopes
resolve_job_prefix()check-runs to the workflow run (including nested reusable workflows), filters to ROCm-only prefixes, and verifies--archbefore auto-healing.Stacked on
ew/parity-cuda-dynamic-shards.Summary
Fixes a bug in
resolve_job_prefix()where unfiltered commit-wide check-runs could cause wrong job prefixes (CUDA or other ROCm arches) to be selected when the configured prefix was stale.referenced_workflowsanddetails_urlexpansion--archbefore auto-healing; fall back to configured prefix on mismatchget_all_check_runs_for_commit()helper to avoid passing empty prefix as a footgunTest plan
python3 -m py_compile .automation_scripts/pytorch-unit-test-scripts/download_testlogsAuthored with assistance from Cursor
Made with Cursor