Skip to content

[CI] Parity: classify newly observed ROCm skips - #3523

Open
ethanwee1 wants to merge 2 commits into
ROCm:developfrom
ethanwee1:ew/parity-cuda-dynamic-shards
Open

[CI] Parity: classify newly observed ROCm skips#3523
ethanwee1 wants to merge 2 commits into
ROCm:developfrom
ethanwee1:ew/parity-cuda-dynamic-shards

Conversation

@ethanwee1

@ethanwee1 ethanwee1 commented Jul 30, 2026

Copy link
Copy Markdown

Summary

  • classify Triton kernel, Triton heuristic, and fused-attention tests as PT2.0 - Inductor
  • classify backend skips, including TVM coverage, as PT2.0 - Dynamo
  • classify the new hipSOLVER DnXsytrs requires ROCm >= 7.14 guard as hipSolver/Magma
  • support both dotted and slash-separated test file names

Dashboard analysis

Reviewed the AI analysis generated on 2026-08-04. It confirms that the inductor.test_triton_heuristics and TVM backend movements are classifier artifacts addressed by this PR. I also checked the underlying analyzed mi350 report rather than relying only on the AI recommendations: all 62 incoming unclassified skips are now covered—60 hipSOLVER version guards and two inductor.test_fused_attention tests.

The analysis also reports large historical convolution, distributed, FSDP, and Misc category movements, but explicitly identifies them as unrelated collection/classifier churn and does not provide test-level evidence for a new category rule. This PR does not add speculative broad rules for those movements.

This PR was split from the earlier parity automation stack and contains only skip-reason classification changes.

Test plan

  • Run Python syntax, lint, and whitespace checks
  • Run the classifier against the analyzed mi350 parity CSV
  • Confirm coverage rises from 62 unclassified skips to 0 (100% of 2,877 target tests classified)
  • Verify this PR changes only auto_classify_skip_reasons.py

@ethanwee1

Copy link
Copy Markdown
Author

Validation runs

Dispatched parity.yml on the fix branch (ethanwee1/pytorch) for SHA 050ce9e1226b95d6b5938bdd5e245491f7219fd8 — the exact SHA that was failing before the fix — across all ROCm arches:

arch run result CUDA shards detected
mi350 30567121048 ✅ success default=14, distributed=10 (25 CUDA jobs)
mi200 30569040561 ✅ success default=14, distributed=10 (25 CUDA jobs)
mi300 30569038602 ⚠️ flagged incomplete default=14, distributed=10 (25 CUDA jobs)

The CUDA baseline now downloads correctly on every arch — no more TEST KEY ... DOES NOT EXIST IN JOBS / no artifacts found on the CUDA side.

The mi300 run is flagged failed only by the intentional graceful-degradation step: for this specific SHA the ROCm mi300 test-report artifacts (default/distributed/inductor) are absent in S3/GHA, so a partial report was produced and the job self-flagged. That is a pre-existing data-availability path unrelated to this change (the CUDA fix itself worked identically there).

@ethanwee1
ethanwee1 force-pushed the ew/parity-cuda-dynamic-shards branch from 4a4d7b1 to 0376854 Compare July 30, 2026 22:31
@ethanwee1

Copy link
Copy Markdown
Author

Rebased onto latest develop (0376854). Ready for Jenkins re-run.

@ethanwee1

Copy link
Copy Markdown
Author

Tested — now detects shard counts for all architectures

Second commit extends dynamic shard detection to every ROCm arch (mi350/mi300/mi200/…) plus the CUDA trunk baseline. The standalone ROCm arch workflows run their shards inside a reusable workflow (workflow_call), so those shards don't appear in the jobs API — derive_shard_count now also consults the commit check-runs API before the config fallback.

Validation runs on the PR branch code (ethanwee1/pytorch, SHA 21f65597):

arch default distributed inductor CUDA run report
mi350 8 3 2 14/10 30654399051 366,939 rows
mi200 10 3 4 14/10 30654402900 360,145 rows
mi300 8 (config had 6) 3 4 (config had 2) 14/10 ⚠️ 30654401126 shards correct; artifacts in trunk-rocm-sandbox (separate follow-up)

mi300 proves the detection catches an actual upstream reshard (6→8, 2→4) that the hardcoded config missed. Its run only fails on the distinct "which run hosts the reusable-workflow artifacts" issue — not a shard-count problem — which is being tracked separately.

@ethanwee1

Copy link
Copy Markdown
Author

Added a third commit: auto-detect ROCm job-name prefixes (resolve_job_prefix). The per-arch prefix in parity_job_config.json (e.g. linux-noble-rocm-py3.12-mi300, the nightly gfx942mi350 rename) is now a self-healing hint — detected from the resolved run's jobs API + commit check-runs when it drifts, kept as-is when still present. CUDA stays configured (trunk carries multiple deliberate CUDA families). Live no-regression mi350 run: 30655560233 ✅ (366,948 rows). Merged to the deployment target ethanwee1/pytorch main as #5.

@jithunnair-amd

Copy link
Copy Markdown
Collaborator

Follow-up fix for resolve_job_prefix scoping: ethanwee1#6 (stacked on this branch). Scopes check-runs to workflow runs, filters ROCm-only prefixes, and adds arch verification.

@ethanwee1

Copy link
Copy Markdown
Author

Ported the two remaining deployment-fork fixes into this PR:

  • Busted job URLs (_wf_run_ids.json per-job run-id map; summarize builds each shard's URL from it). Matches fork #7.
  • Artifact-hosting run (resolve_artifact_download: ROCm arch shards that run under trunk-rocm-sandbox are downloaded from the run that actually ran them, filtered by exact job ids). Matches fork #8.

This PR now carries the full set: CUDA shard detection, all-arch ROCm shard detection (check-runs), job-name prefix auto-detection, busted-job-URL fix, and artifact-hosting-run resolution.

Acting on the HUD AI-analysis, which flagged commit-to-commit skip-category
churn caused by newly-added upstream tests that lacked a classification rule
(so they landed in the wrong / blank bucket instead of their real category).

Add file-based rules for three unambiguous suites that had no rule:
  - inductor/test_triton_kernels   -> PT2.0 - Inductor
  - inductor/test_triton_heuristics-> PT2.0 - Inductor
  - dynamo/test_backends (TVM)     -> PT2.0 - Dynamo

Rules match both '.' and '/' test_file separators for robustness. Verified
they classify correctly and leave unrelated files (e.g. test_torch -> Misc)
untouched.
@ethanwee1
ethanwee1 force-pushed the ew/parity-cuda-dynamic-shards branch from 340abeb to 4159f02 Compare August 3, 2026 20:23
@ethanwee1 ethanwee1 changed the title [CI] Parity: derive CUDA shard counts from job names (fix all-runs-failing reshard) [CI] Parity: classify Triton and TVM backend skips Aug 3, 2026
Route the new hipSOLVER version guard and fused-attention tests into specific categories so the current dashboard report has no unclassified parity skips.
@ethanwee1 ethanwee1 changed the title [CI] Parity: classify Triton and TVM backend skips [CI] Parity: classify newly observed ROCm skips Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants