Add therock by ethanwee1 · Pull Request #2 · ethanwee1/pytorch

ethanwee1 · 2025-11-21T19:19:42Z

Fixes #ISSUE_NUMBER

…e using eval frame overrides (pytorch#173080) This is attempt #2 at pytorch#172295. Instead of using frame/CacheEntry recursive actions, we override the recursive eval_frame callback in eval_frame_cpp.cpp if we run custom code. The override is set in eval_frame.py only if fullgraph=True. Right now, the possible overrides are: - skip tracing frames - error if tracing is successful. We can't just error on entry since convert_frame might end up skipping the frame, which should be allowed. It might be better to simply patch the entrypoint of tracing e.g. InstructionTranslatorBase, with an error though. Currently, the default behavior is the latter, since we should loudly fail if torch.compile gets unintentionally re-invoked when fullgraph=True. Pull Request resolved: pytorch#173080 Approved by: https://github.com/anijain2305

If another static object (like `g_device_config_parse_hook_registry_instance` created by the `REGISTER_ALLOCATOR_CONFIG_PARSE_HOOK` macro) tries to call `registerDeviceConfigParserHook` before `device_config_parser_hook_` is initialized, assigning to it (operator=) can fail, which leads to a runtime error. When I use a compilation optimization of ` -O1` I see this issue: ``` [src/libcxx/include/__functional/function.h:496]:14: runtime error: member access within null pointer of type 'const __policy' #0 0x563224e28b78 in operator= [crosstool/v18/stable/src/libcxx/include/__functional/function.h:496]:14 #1 0x563224e28b78 in operator= [crosstool/v18/stable/src/libcxx/include/__functional/function.h:483]:19 #2 0x563224e28b78 in operator= [crosstool/v18/stable/src/libcxx/include/__functional/function.h:727]:8 #3 0x563224e28b78 in c10::CachingAllocator::AcceleratorAllocatorConfig::registerDeviceConfigParserHook(std::__u::function<void (std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>> const&)>&&, std::__u::unordered_set<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>, std::__u::hash<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>>, std::__u::equal_to<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>>, std::__u::allocator<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>>> const&) [torch/c10/core/AllocatorConfig.h:263]:32 #4 0x563224e28e9d in DeviceConfigParserHookRegistry [torch/c10/core/AllocatorConfig.h:369]:5 #5 0x563224e28e9d in __cxx_global_var_init.34 [torch/c10/cuda/CUDAAllocatorConfig.cpp:195]:1 #6 0x563224e28e9d in _GLOBAL__sub_I_CUDAAllocatorConfig.cpp torch/c10/cuda/CUDAAllocatorConfig.cpp #7 0x5632459709ac in __libc_csu_init /[usr/grte/v5/debug-src/src/csu/elf-init.c:88]:7 #8 0x7f748b9562e7 in __libc_start_main (/usr/grte/v5/lib64/libc.so.6+0x612e7) (BuildId: ca23ec6d935352118622ce674a8bb52d) #9 0x5632018f3729 in _start /usr/grte/v5/debug-src/src/csu/../sysdeps/x86_64/start.S:120 ``` Pull Request resolved: pytorch#172581 Approved by: https://github.com/guangyey, https://github.com/albanD

…rch#175067) ## Summary Move CUDA 12.8 GPU tests from per-commit trunk CI to periodic (~3x/day on weekdays). Both CUDA 12.8 and 13.0 are shipping wheel targets (nightly ships cu126, cu128, cu129, cu130), but their trunk CI test suites have **85-90% failure correlation** -- they almost always fail together. Over a 30-day analysis window covering 97 reverts and 38 significant regression events, **CUDA 12.8 never uniquely caught a regression that 13.0 missed**. CUDA 13.0 is kept per-commit because: - It is the **newest** shipping CUDA version - Most likely to surface **novel breakage** from new CUDA runtime behavior - Forward-looking CI should protect what's coming, not what's already stable CUDA 12.8 is moved to periodic because: - It is **mature and well-understood** -- breakage is less likely and less urgent - The rare 12.8-only regression can tolerate the ~8-hour periodic detection window - The 12.8 build job **remains in trunk** because `cross-compile-linux-test` depends on its artifacts **Estimated savings: ~1,270 GPU-hours/week (~5,080 GPU-hours/month)** This is the #2 savings opportunity from a broader CI workflow analysis (P2188981399) covering 128 PR+trunk jobs over 30 days. Combined with pytorch#175066 (CycleGAN skip, ~310 GPU-hours/week), total savings from this stack: **~1,580 GPU-hours/week (~6,320 GPU-hours/month)**. ### Changes - `trunk.yml`: remove CUDA 12.8 test job (5 default + 3 distributed + 1 pr_time_benchmarks + 1 libtorch shards) and no-ops build - `periodic.yml`: add default (5 GPU shards on g6.4xlarge) and distributed (3 multi-GPU shards on g4dn.12xlarge) to existing CUDA 12.8 periodic entry ## Test Plan - CUDA 12.8 GPU tests continue to run in periodic (3x/day weekdays) - CUDA 13.0 per-commit coverage is unchanged - Cross-compile-linux-test continues to work (12.8 build job kept) Pull Request resolved: pytorch#175067 Approved by: https://github.com/malfet ghstack dependencies: pytorch#175066

…rch#175300) [CI] Move CUDA 12.8 GPU tests from per-commit trunk to periodic (pytorch#175067) ## Summary Move CUDA 12.8 GPU tests from per-commit trunk CI to periodic (~3x/day on weekdays). Both CUDA 12.8 and 13.0 are shipping wheel targets (nightly ships cu126, cu128, cu129, cu130), but their trunk CI test suites have **85-90% failure correlation** -- they almost always fail together. Over a 30-day analysis window covering 97 reverts and 38 significant regression events, **CUDA 12.8 never uniquely caught a regression that 13.0 missed**. CUDA 13.0 is kept per-commit because: - It is the **newest** shipping CUDA version - Most likely to surface **novel breakage** from new CUDA runtime behavior - Forward-looking CI should protect what's coming, not what's already stable CUDA 12.8 is moved to periodic because: - It is **mature and well-understood** -- breakage is less likely and less urgent - The rare 12.8-only regression can tolerate the ~8-hour periodic detection window - The 12.8 build job **remains in trunk** because `cross-compile-linux-test` depends on its artifacts **Estimated savings: ~1,270 GPU-hours/week (~5,080 GPU-hours/month)** This is the #2 savings opportunity from a broader CI workflow analysis (P2188981399) covering 128 PR+trunk jobs over 30 days. Combined with pytorch#175066 (CycleGAN skip, ~310 GPU-hours/week), total savings from this stack: **~1,580 GPU-hours/week (~6,320 GPU-hours/month)**. ### Changes - `trunk.yml`: remove CUDA 12.8 test job (5 default + 3 distributed + 1 pr_time_benchmarks + 1 libtorch shards) and no-ops build - `periodic.yml`: add default (5 GPU shards on g6.4xlarge) and distributed (3 multi-GPU shards on g4dn.12xlarge) to existing CUDA 12.8 periodic entry ## Test Plan - CUDA 12.8 GPU tests continue to run in periodic (3x/day weekdays) - CUDA 13.0 per-commit coverage is unchanged - Cross-compile-linux-test continues to work (12.8 build job kept) Pull Request resolved: pytorch#175067 Approved by: https://github.com/malfet ghstack dependencies: pytorch#175066 (cherry picked from commit ef0353f) Co-authored-by: Eli Uriegas <eliuriegas@meta.com>

…nces between x86 vs aarch64 (pytorch#176085) In the test: ``` python test/cpp_extensions/test_libtorch_agnostic.py TestLibtorchAgnosticCUDA.test_std_cuda_check_error_show_cpp_stacktraces_True_cuda ``` it raises an exception when calling `STD_CUDA_CHECK(cudaSetDevice(99999));` which got the expected `CUDA error: invalid device` message. However, the expected string for the C++ stack trace is different between `x86` vs `aarch64` due perhaps in these issues: - pytorch#119905 - pytorch#134387 In the current setup when getting a stack trace string: - x86 contains `C++ CapturedTraceback:` - aarch64 contains `Exception raised from` + `frame #` An example of the full string from an aarch64 system when : ``` AssertionError: 'C++ CapturedTraceback:' not found in 'CUDA error: invalid device ordinal\nGPU device may be out of range, do you have enough GPUs?\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n\nException raised from test_std_cuda_check_error at /opt/pytorch/pytorch/test/cpp_extensions/libtorch_agn_2_10_extension/csrc/test_std_cuda_check.cu:23 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xd4 (0xe471ebcd39f4 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)\nframe #1: <unknown function> + 0x43f998 (0xe471ebdcf998 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, unsigned int, bool) + 0x1bc (0xe471ebdcfc0c in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)\nframe #3: torch_c10_cuda_check_msg + 0x1c (0xe471ef335c4c in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\nframe #4: test_std_cuda_check_error() + 0x58 (0xe470cd396678 in /opt/pytorch/pytorch/test/cpp_extensions/libtorch_agn_2_10_extension/install/usr/local/lib/python3.12/dist-packages/libtorch_agn_2_10/_C.so)\nframe #5: c10::BoxedKernel::makeFromFunctor<StableIValueBoxedKernel>(std::unique_ptr<StableIValueBoxedKernel, std::default_delete<StableIValueBoxedKernel> >)::{lambda(c10::OperatorKernel*, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocator<c10::IValue> >*)#1}::_FUN(c10::OperatorKernel*, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocator<c10::IValue> >*) + 0x16c (0xe47211cd419c in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x61d34bc (0xe47211cf34bc in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)\nframe #7: <unknown function> + 0xe6c324 (0xe4721532c324 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)\nframe #8: <unknown function> + 0xe6c7e0 (0xe4721532c7e0 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)\nframe #9: <unknown function> + 0xd3907c (0xe472151f907c in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)\nframe pytorch#10: <unknown function> + 0x5ccbf8 (0xe47214a8cbf8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)\nframe pytorch#11: /usr/bin/python() [0x504a34]\nframe pytorch#12: PyObject_Call + 0x6c (0x4c633c in /usr/bin/python)\nframe pytorch#13: _PyEval_EvalFrameDefault + 0x3ea0 (0x568564 in /usr/bin/python)\nframe pytorch#14: _PyObject_Call_Prepend + 0xc4 (0x4c5934 in /usr/bin/python)\nframe pytorch#15: /usr/bin/python() [0x52a070]\nframe pytorch#16: _PyObject_MakeTpCall + 0x78 (0x4c3e58 in /usr/bin/python)\nframe pytorch#17: _PyEval_EvalFrameDefault + 0x8a0 (0x564f64 in /usr/bin/python)\nframe pytorch#18: PyEval_EvalCode + 0x130 (0x5632b4 in /usr/bin/python)\nframe pytorch#19: PyRun_StringFlags + 0xe0 (0x59c330 in /usr/bin/python)\nframe pytorch#20: PyRun_SimpleStringFlags + 0x44 (0x67ebc4 in /usr/bin/python)\nframe pytorch#21: Py_RunMain + 0x390 (0x68b380 in /usr/bin/python)\nframe pytorch#22: Py_BytesMain + 0x28 (0x68ae88 in /usr/bin/python)\nframe pytorch#23: <unknown function> + 0x284c4 (0xe47216b084c4 in /lib/aarch64-linux-gnu/libc.so.6)\nframe pytorch#24: __libc_start_main + 0x98 (0xe47216b08598 in /lib/aarch64-linux-gnu/libc.so.6)\nframe pytorch#25: _start + 0x30 (0x5f6770 in /usr/bin/python)\n\n' To execute this test, run the following from the base repo dir: python test/cpp_extensions/test_libtorch_agnostic.py TestLibtorchAgnosticCUDA.test_std_cuda_check_error_show_cpp_stacktraces_True_cuda ``` Pull Request resolved: pytorch#176085 Approved by: https://github.com/eqy

… mode + replication padding (pytorch#177166) Fixes pytorch#170079 ## Context `torch.compile(ReplicationPad1d(...), fullgraph=True)` crashes when `torch.use_deterministic_algorithms(True)` is set on CUDA. The error: Dynamo can't trace through `importlib.import_module`. The deterministic code path exists because the native `replication_pad1d_backward` CUDA kernel uses `atomicAdd` (non-deterministic). `functional.py` calls `_replication_pad` — a Python decomposition using `_unsafe_index`, whose backward uses `index_put` (deterministic). ## Dynamo limitations encountered Three separate Dynamo tracing barriers prevented calling `_replication_pad` directly: ### 1. `importlib.import_module` is marked as skipped ```python @torch.compile(fullgraph=True) def fn(x): import importlib return importlib.import_module("torch").sin(x) fn(torch.randn(3)) # Unsupported: function marked as skipped ``` ### 2. `elementwise_dtypes` returns non-Tensor (from `@pw_cast_for_opmath`) ```python @torch.compile(fullgraph=True) def fn(x): from torch._prims_common import elementwise_dtypes, ELEMENTWISE_TYPE_PROMOTION_KIND dt, _ = elementwise_dtypes(x, type_promotion_kind=ELEMENTWISE_TYPE_PROMOTION_KIND.DEFAULT) return x.to(dt) fn(torch.randn(3)) # Unsupported: torch.* op returned non-Tensor ``` ### 3. `torch._check` with closure lambda ```python @torch.compile(fullgraph=True) def fn(x): dim = x.dim() torch._check(dim in (2, 3), lambda: f"expected 2D or 3D, got {dim}D") return x + 1 fn(torch.randn(3, 3)) # Unsupported: Can't extract message from torch._check() ``` ## Iteration log | # | Approach | Who | Tests | Reviewer pushback | Why it failed | |---|----------|-----|-------|-------------------|---------------| | 1 | Replace `importlib` with `from...import` | Claude | bilinear/trilinear pass, replicate fails | "why do we need bilinear/trilinear tests?" — scoped fix to reported bug only | Hit limitation #2: `@pw_cast_for_opmath` | | 2 | Skip decomposition under compile via `is_compiling()`, rely on AOTAutograd's `@register_decomposition` | Claude | forward-only `backend="eager"` passes | "can you verify at inductor level this is actually deterministic?" — inspect AOT graph | No backward decomposition registered; backward still uses native `replication_pad1d_backward` (non-deterministic) | | 3 | Unwrap `@pw_cast_for_opmath` via `__wrapped__` | Claude | N/A — fails immediately | N/A | Hit limitation #3: `torch._check()` closure | | 4 | `@nonstrict_trace` — Dynamo skips body, AOTAutograd traces through | Reviewer suggestion | `backend="aot_eager"`, forward + backward under `DeterministicGuard(True)` | N/A — fix is correct | N/A | ## Key insight The fix isn't about making Dynamo trace the decomposition or skipping it entirely — it's about putting the boundary in the right place. Dynamo doesn't need to see inside; AOTAutograd does. `@nonstrict_trace` is exactly this boundary. Each "obvious" fix had passing tests that weren't testing the right thing. Only when the reviewer pushed for backward determinism verification and AOT graph inspection did the weaknesses surface. The backward completing without error under `DeterministicGuard(True)` proves determinism — PyTorch explicitly raises `RuntimeError` if any non-deterministic CUDA kernel executes under this mode. Authored with Claude. Pull Request resolved: pytorch#177166 Approved by: https://github.com/mlazos, https://github.com/williamwen42

…#188024) Some gfx950 (MI350) 2-GPU runner pods come up with a container that cannot read part of the KFD/HSA topology. RCCL then fails every collective init with "ncclUnhandledCudaError: Call to CUDA function failed / Could not read node #N" (N is a fixed topology-node index for that pod, e.g. #2 or #9). When a distributed shard lands on such a pod, the first collective crashes and a later test hangs the whole shard until the 270-minute job timeout. Host-side rocminfo still enumerates the GPUs on these pods, so the existing "Runner check GPU count" gate does not catch it -- the failure is the container's deeper topology read, not agent enumeration. Diagnosis: the same RCCL error appears across many different distributed tests and both worker-crash and downstream-hang forms; in the logs it is constant per pod and present from the very first collective, and world_size is 2 so "node #N" cannot be a rank -- it is a system topology node. So it is a per-pod container health problem, not a PyTorch or per-test bug. Fix: add a fast in-container RCCL pre-flight (.ci/pytorch/rocm_preflight.py) that spawns a min(2, ngpu)-rank process group and does one all_reduce. For distributed shards _rocm-test.yml runs it (wrapped in `timeout 180`) before the suite; on failure or hang the job fails in seconds with a clear message instead of hanging for 270 minutes, and the bad pod is identifiable for draining. Non-distributed shards are unchanged. Test Plan: ``` python -m py_compile .ci/pytorch/rocm_preflight.py lintrunner -a .ci/pytorch/rocm_preflight.py .github/workflows/_rocm-test.yml ``` Validated the pre-flight directly. Locally on 8x A100 (the nccl backend exercises the same init/collective path RCCL uses) the script passes: ROCm/RCCL pre-flight passed (2-rank all_reduce) and the failure path is exercised too: when the process group cannot initialize, mp.spawn raises, the script prints the "::error::" message and exits 1 -- the job fails fast instead of hanging. In CI, dispatched the gate on the gfx950.2 distributed pool: the pre-flight ran inside the container and passed on healthy pods (exit 0, suite proceeded). The broken-pod case is the same exit-1 fast-fail path; it only reproduces when a pod with the broken KFD topology is in rotation, and the pool was healthy during testing, so that specific case was not caught live. Authored with Claude. Pull Request resolved: pytorch#188024 Approved by: https://github.com/frgossen

…oad_rocm_deps`) (pytorch#188454) ## Summary This PR makes PyTorch ROCm/TheRock wheels **self-contained**: it fixes the two `import torch` failures that occur on the wheel (`_rocm_sdk_*` site-packages) layout, so `import torch` works without `LD_LIBRARY_PATH` or a sourced ROCm env (e.g. the ROCm distributed preflight `python .ci/pytorch/rocm_preflight.py`, which never sources `/etc/rocm_env.sh`). Both fixes live in torch **source**, so they travel with any build (including a bare `python -m build` in a TheRock env), not just CI-produced wheels. This is the **minimal, consolidated PR** carrying **both** import-discovery fixes — a **two-file** diff (`torch/__init__.py` + `torch/cuda/__init__.py`). Both changes are **no-ops on the traditional tarball `/opt/rocm` install**: each early-returns when the wheel package's `_rocm_sdk_core` is absent, so non-wheel ROCm CI (apt/`/opt/rocm`, whose libs are already on the loader path), CUDA, and CPU builds are unaffected. ## Fixes ### 1. `torch/__init__.py` — `_preload_rocm_deps()` Fixes `ImportError: librocm_sysdeps_liblzma.so.5: cannot open shared object file` at `from torch._C import *` - Adds `_preload_rocm_deps()`, the ROCm analogue of `_preload_cuda_deps()` (which handles the `nvidia-*-cu12` wheels). At import time, **before** `import torch._C`, it locates the `rocm` pip package's `_rocm_sdk_core` via `importlib.util.find_spec` (the same anchor `torch.utils.cpp_extension._find_rocm_home()` uses) and `ctypes.CDLL`-preloads (`RTLD_GLOBAL`) the vendored sysdeps (`librocm_sysdeps_*`) and core ROCm runtime libs from `_rocm_sdk_core/lib` and `_rocm_sdk_core/lib/rocm_sysdeps/lib`. Wired into `_load_global_deps()`. - Guarded to no-op on non-Linux, on non-ROCm builds (`torch.version.hip is None` → CUDA/CPU), and on OS-managed ROCm where `_rocm_sdk_core` is absent (the apt/`/opt/rocm` path). ### 2. `torch/cuda/__init__.py` — amdsmi / `libamd_smi.so` discovery hook + non-fatal guard Fixes the *next* failure that surfaces once `_preload_rocm_deps()` gets `import torch` past the `librocm_sysdeps` error: `KeyError: 'libamd_smi.so'` raised from the `amdsmi` Python binding at `import amdsmi` (inside `_C._initExtension`) On TheRock wheels the amdsmi package installs the native library as a versioned soname under `_rocm_sdk_core/lib` (e.g. `libamd_smi.so.26`), which amdsmi's own `find_smi_library()` fails to locate when `$ROCM_HOME`/`$ROCM_PATH` and the loader path are unset; its wrapper then does `_libraries['libamd_smi.so']` → `KeyError`, which escaped the existing guard (only `ModuleNotFoundError` was tolerated) and killed `import torch`. Two-part torch-side hardening: - Make `_amdsmi_cdll_hook` **TheRock-aware**: locate `_rocm_sdk_core` via `importlib.util.find_spec` and add its `lib/libamd_smi.so*` (glob, incl. versioned sonames) as last-resort candidate paths; match names whose basename **starts with** `libamd_smi.so` so amdsmi's bare `CDLL("libamd_smi.so")` is redirected to the real versioned file. Existing `ROCM_HOME`/`ROCM_PATH` behavior preserved; guarded to Linux. - Make a non-discoverable amdsmi **non-fatal**: broaden the guard so `KeyError`/`OSError` degrade to `_HAS_AMDSMI=False` instead of aborting `import torch`, matching the existing missing-module path. ## Consolidation note The amdsmi discovery fix (#2 above) originally shipped as standalone PR pytorch#188724. That PR has been **closed** and folded into this PR so **both** import-discovery fixes travel together in a single minimal two-file change. Scope note: the TheRock tarball→wheels migration and the earlier CI-only RPATH/ldconfig workarounds are **not** part of this PR. The wheels migration is tracked separately by pytorch#188429, which rebases on top of this change. ## Validation (TheRock wheel path) — PASSED Validated end-to-end by **temporarily** stacking Ethan's wheels-migration PR pytorch#188429 (which adds the `rocm-preview` wheel CI) on top of the two source fixes and running `ciflow/rocm-preview` (run [28581558341](https://github.com/pytorch/pytorch/actions/runs/28581558341)) on the stacked head. The temporary stack was then **removed**, so this PR remains the minimal two-file source diff. - **`build-osdc` (TheRock wheel-path build): SUCCESS** — [job 84742887851](https://github.com/pytorch/pytorch/actions/runs/28581558341/job/84742887851). - **`import torch` works on the `_rocm_sdk_*` wheel layout** — proven by the passing shard [`test (default, 5/6)` job 84772673709](https://github.com/pytorch/pytorch/actions/runs/28581558341/job/84772673709), which imports torch and prints the HIP 7.14 / MIOpen 3.5.2 build config. - **No** `ImportError: librocm_sysdeps*` (incl. `librocm_sysdeps_liblzma`) and **no** `KeyError: 'libamd_smi.so'` anywhere in the run → **both** fixes are validated on the **real TheRock wheel path**. - Remaining rocm-preview shard failures were **unrelated** to this PR — test-content failures (`test_CTCLoss_cudnn_cuda`, `test_cxx_flags` `RuntimeError map::at`, `test_hip_device_count`, `test_lazy_imports_are_lazy`) and infra (`ROCm pre-flight: no GPUs visible to the container` on the distributed shards). - Run: https://github.com/pytorch/pytorch/actions/runs/28581558341 ### rocm-preview distributed shards (register fat binary failed) Reference/evidence for the `register fat binary failed` (`code_object.cpp:1038`) investigation — the rocm-preview (TheRock wheel-path) distributed shards from run [28581558341](https://github.com/pytorch/pytorch/actions/runs/28581558341) (`linux-noble-rocm-preview-py3.12-gfx942`), all FAILED: - distributed 1/3: https://github.com/pytorch/pytorch/actions/runs/28581558341/job/84772673683 (`test (distributed, 1, 3, linux.rocm.gpu.gfx942.4, module:rocm, oncall:distributed)`) - distributed 2/3: https://github.com/pytorch/pytorch/actions/runs/28581558341/job/84772673678 (`test (distributed, 2, 3, linux.rocm.gpu.gfx942.4, module:rocm, oncall:distributed)`) - distributed 3/3: https://github.com/pytorch/pytorch/actions/runs/28581558341/job/84772673727 (`test (distributed, 3, 3, linux.rocm.gpu.gfx942.4, module:rocm, oncall:distributed)`) <details> <summary>Failure log the amdsmi fix addresses (pre-fix, run 28524549808)</summary> ``` /var/lib/jenkins/workspace/libamd_smi.so: cannot open shared object file: No such file or directory Unable to find libamd_smi.so library try installing amd-smi-lib from your package manager Traceback (most recent call last): File "/var/lib/jenkins/workspace/.ci/pytorch/rocm_preflight.py", line 21, in <module> import torch File ".../site-packages/torch/cuda/__init__.py", line 114, in <module> import amdsmi # type: ignore[import] File ".../site-packages/amdsmi/amdsmi_wrapper.py", line 218, in <module> amdsmi_free_name_value_pairs = _libraries['libamd_smi.so'].amdsmi_free_name_value_pairs KeyError: 'libamd_smi.so' ``` </details> ## Notes: amdsmi discovery + fat-binary logs - **Bare `CDLL("libamd_smi.so")`** comes from amdsmi's own `find_smi_library()` (unversioned candidate), not torch; torch's `_amdsmi_cdll_hook` intercepts it and retries the versioned `_rocm_sdk_core/lib/libamd_smi.so*`. - **Not just preload amdsmi via `_rocm_core_libs`:** amdsmi resolves by exact filename, so an RTLD_GLOBAL versioned `libamd_smi.so.26` won't satisfy it, and eager-preloading a non-`DT_NEEDED` lib reintroduces the ODR double-load the hook prevents — the CDLL-redirect hook is the correct fix. - **`E0702 … code_object.cpp:1038 register fat binary failed` is not caused by this PR:** it's emitted by the HIP/clr C++ runtime, appears ambiently (~4,774×/run across rocm-preview gfx942 jobs incl. the distributed shards above), and is non-fatal — a pre-existing TheRock rocm-preview build/toolchain matter, orthogonal to these two Python-only changes. - **Fix-plan pointer (fat-binary):** prove pre-existing on a PR-free `main` rocm-preview baseline → localize via `AMD_LOG_LEVEL=3/4` → scoped clr-log suppress/demote (escalate to clr/comgr pin or `PYTORCH_ROCM_ARCH` only on a real running-arch failure) → validate via `ciflow/rocm-preview`. Belongs in the build/CI layer, not torch source. Tracked in ROCm/frameworks-internal pytorch#17160. ## Test plan - [x] Validate on the TheRock wheel path via `ciflow/rocm-preview` (see **Validation** above). - [x] `ast.parse` both changed files (`torch/__init__.py`, `torch/cuda/__init__.py`). - [x] Confirm CUDA and CPU builds are unaffected (both fixes no-op when `torch.version.hip is None` / `_rocm_sdk_core` absent). Fixes pytorch#188561 Made with Cursor Pull Request resolved: pytorch#188454 Approved by: https://github.com/jeffdaily

…input (pytorch#191002) **LLM Statement**: This is in most part the work of my Claude, both the bug diagnostic, the narrowing down to a minimal reprod, the suggestion of a fix, and the body of this PR where done by it. However, I did observe the bug, and this fix does solve it. I don't personally know the internals enough to understand if this is the right fix, but it looks quite worthwhile to me. Thanks in advance for reviewing! ## Summary Fixes a silent-correctness bug in `torch.compile` where an output that aliases a compiled function's input can point at the **wrong slice** of that input when inductor's alignment path has cloned the input onto a fresh storage. Root cause: `torch/_functorch/_aot_autograd/functional_utils.py::gen_alias_from_base` uses `target_meta_tensor.storage_offset()` (the traced FakeTensor's offset, which is `0` when the target lives on a cloned aligned storage) directly as the offset into the *original* aliased base tensor. That picks the start-of-storage slice instead of the slice at the input's real `storage_offset`. ## Minimal pure-PyTorch reproducer (2 GPUs, `torchrun --nproc_per_node=2`) ```python import os, torch, torch.distributed as dist import torch.distributed.fsdp as fsdp, torch.nn as nn, torch.nn.functional as F R, W = int(os.getenv("RANK")), int(os.getenv("WORLD_SIZE")) torch.cuda.set_device(R) dist.init_device_mesh("cuda", (W,)) D, S = 512, 512 def _sl(x): return x if W == 1 else x.chunk(W)[R] class B(nn.Module): def __init__(s): super().__init__() s.i = nn.Linear(D, D, bias=False) def forward(s, x): return s.i(x) class M(nn.Module): def __init__(s): super().__init__() s.e = nn.Embedding(D, D) s.b = nn.ModuleList([B()]) def forward(s, tok, tgt): _sl(tok.new_zeros(W)) h = s.e(_sl(tok.flatten())).view(1, -1, D) for blk in s.b: h = blk(h) return F.cross_entropy(h.reshape(-1, D), _sl(tgt.flatten()), reduction="none") torch.manual_seed(0) m = M().cuda() for p in m.parameters(): p.data.normal_(0, 0.02) for b in m.b: fsdp.fully_shard(b) m.compile() torch.manual_seed(1) r = ((torch.rand(1, S + 1) ** 8) * D).long().cuda() loss = m(r[:, :-1], r[:, 1:]).float().mean() loss.backward() ``` `tok = r[:, :-1]` and `tgt = r[:, 1:]` share `r`'s storage at offsets 0 and 1. `tgt.data_ptr()` lands at `r.data_ptr() + 8`, which is misaligned to 16 bytes, so inductor's `copy_misaligned_inputs` clones it onto a fresh aligned buffer. When the compiled forward returns an alias of `tgt` (via `x.chunk(W)[R]` on it), `gen_alias_from_base` gets `target_meta_tensor.storage_offset() == 0` (the clone's offset) and applies it to the original `r`. Result: the alias points at `r[:, :-1]` (the `tok` slice), not at `r[:, 1:]` — so `F.cross_entropy` computes loss against the input tokens shifted by one, not the labels. ## Non-distributed reproducer (now the in-tree regression test) The distributed machinery above turned out to be incidental: FSDP2/DTensor/2-ranks only served to (a) hand the compiled graph a misaligned slice and (b) shape the graph so an `alias_of_input` output is regenerated through the `as_strided` fallback of `gen_alias_from_base`. Both are triggerable single-process: ```python def fn(x): return x + 1, x.chunk(2)[1] fn_c = torch.compile(fn) a = torch.randn(1024, device="cuda") torch._dynamo.mark_dynamic(a, 0) # symbolic ViewMeta => as_strided fallback fn_c(a) # compile with an ALIGNED input y = torch.randn(1025, device="cuda")[1:] # misaligned view; offset not guarded on _, out = fn_c(y) # inductor clones y at runtime # before this PR: out is the WRONG slice (out.data_ptr() != y.chunk(2)[1].data_ptr()) ``` Two variants are added as `test/inductor/test_alignment.py::GPUTests::test_alias_of_misaligned_input{,_no_view_replay}`: one reaches the `as_strided` fallback via a symbolic ViewMeta (`chunk` on a `mark_dynamic` dim), the other via `view_replay_for_aliased_outputs=False` (the fbcode default, where the fallback is the main path for aliased outputs). Both fail before the fix and pass with it. The oracle is exact (`data_ptr`/`storage_offset` of the returned alias), not a loss comparison. ## Numbers on the minimal repro Nightly `2.14.0.dev20260722+cu130`: | Configuration | loss | grad `e.weight` | grad `b.0.i.weight` | |-------------------------------------------|----------|-----------------|---------------------| | eager | 6.239221 | 0.0863 | 0.0888 | | compile, **BEFORE** patch | 6.242074 | 0.1992 | 0.0807 | | compile, **AFTER** patch (this PR) | 6.239221 | 0.0863 | 0.0888 | | compile, before patch, `.clone()` inputs* | 6.239221 | 0.0863 | 0.0888 | *Cloning the inputs at the call site sidesteps the bug because the resulting tensors are 16-byte aligned and inductor's clone path is never entered. ## Real-world reproduction and verification (Meta's amaia repo) The bug was originally reported inside a hybrid GDN Qwen3.5 pretraining run (dp=16, tp=1, cp=2, whole-model `torch.compile`) as a 2.41× gradient-norm inflation vs the cp=1 baseline. Reproduced at 2-GPU scale on a shrunk GDN config (n_layers=2, dim=2048, ffn=6144, n_heads=16, whole-model compile, cp=2): | Configuration | grad_norm | vs cp=1 whole-compile | Verdict | |-----------------------------------------------|-----------|-----------------------|---------| | cp=1 whole-compile | 11.8739 | 1.000× | reference | | cp=2 eager | 11.6891 | 0.984× | reference (small CP noise) | | **cp=2 whole-compile, BEFORE patch** | 15.7234 | **1.324× (BUG)** | INFLATED | | cp=2 whole-compile, AFTER patch (this PR) | 11.6888 | 0.984× | FIXED | | cp=2 whole-compile, with a call-site `.clone()` on tokens[:,:-1]/tokens[:,1:] | 11.6888 | 0.984× | FIXED | | cp=2 whole-compile, patch + call-site clone | 11.6888 | 0.984× | FIXED (no interference) | Bit-identical across the three fix variants — this PR and a user-code `.clone()` at the aliased-input construction site address the same root cause independently. Inflation ratio scales with depth: nl=2 → 1.32×, Sergei's real 9B nl=32 → 2.41×. Consistent with a per-layer factor compounding through the aliased-input path in the backward. ## Fix Detect the storage mismatch (which uniquely identifies the "input was cloned by inductor" case) and translate the offset by the runtime input's `storage_offset()`. Complex-view fallbacks reuse the same corrected offset. ```diff --- a/torch/_functorch/_aot_autograd/functional_utils.py +++ b/torch/_functorch/_aot_autograd/functional_utils.py @@ -383,6 +383,20 @@ def gen_alias_from_base( size = target_meta_tensor.size() stride = target_meta_tensor.stride() storage_offset = target_meta_tensor.storage_offset() + # If the target lives on a different storage than the aliased base + # (e.g. because inductor's copy_misaligned_inputs cloned the input to + # obtain an aligned buffer), target.storage_offset() is expressed in + # the cloned storage and would pick the wrong slice when applied via + # as_strided() on the original aliased base tensor. Translate the + # offset: the traced FakeTensor's storage_offset equals the trace-time + # RELATIVE offset from the input, so add back the runtime input's + # storage_offset to keep the alias anchored to the correct slice. + # Compare storages via _cdata (raw c10::Storage handle) rather than + # .data_ptr() so this is safe on fake/meta storages that would raise + # from .data_ptr() during AOT tracing. + if ( + aliased_base_tensor.untyped_storage()._cdata + != target_meta_tensor.untyped_storage()._cdata + ): + storage_offset = aliased_base_tensor.storage_offset() + storage_offset if aliased_base_tensor.is_complex() and not target_meta_tensor.is_complex(): aliased_out = torch.view_as_real(aliased_base_tensor).as_strided( size, stride, storage_offset ``` Note on the storage comparison: an earlier revision compared `untyped_storage().data_ptr()`, which crashes at trace time — `gen_alias_from_base` is also called with FunctionalTensors and traceable wrapper subclasses (e.g. in the synthetic-base path and subclass tests), whose storages raise "Attempted to access the data pointer on an invalid python storage". Comparing `_cdata` (StorageImpl identity, the same mechanism `StorageWeakRef` uses) never dereferences the data pointer. This is what fixed the 8 `test_aotdispatch.py -k alias` failures from the first CI run. The same patch applies verbatim to torch 2.11.0 (backport confirmed by dropping it into amaia's env and re-running the reproducer — identical bit-perfect fix). ## What actually needs to happen for the bug to fire Two independent conditions: 1. **Misaligned runtime input.** An input tensor's `data_ptr()` must not be 16-byte aligned. CUDA's caching allocator lands base allocations on multiples of 512, so this is trivially triggered by any int64/float32/float64 view at `storage_offset != 0` on a tensor that was itself the start of an allocation: e.g. `tokens[:, 1:]` where `tokens` was freshly allocated. 2. **Compiled forward returns an `alias_of_input` output rooted at that input.** AOT only calls `gen_alias_from_base` on outputs that ViewAndMutationMeta marks as aliasing an input. Whether an output is marked that way depends on the FX partition, not directly on the user code — different graph shapes surface different aliases. Both conditions can be true 100% of the time (fully deterministic) but the bug still only surfaces on graphs that have condition #2. That's why the bug is easy to see in specific configurations (like the minimal pure-PyTorch repro above and the amaia GDN reproducer) but doesn't show up in every FSDP2 + torch.compile setup — small dense-attention transformers and small hybrid-GDN transformers that were tested during triage did NOT reproduce it, because their graphs happen not to emit an `alias_of_input` output rooted in a misaligned input tensor. ## Minimum ingredients that produce condition #2 in the pure-PyTorch repro Ablation on the minimal repro (10 variants, each removing one thing, verified in a reverted-patch env with 3 trials per variant): - **2 ranks** (`dist.init_device_mesh` with W=2) — single-rank collapses `_sl` to identity, no alias output. - **`fully_shard` on a nested block** — the DTensor subclass boundary crossing the compile boundary is what causes AOT to *surface* the aliased int-index tensor as a graph output. Removing FSDP2 keeps that value internal to saved-for-backward. - **`nn.ModuleList` + `for blk in self.b:` loop** — indexed `self.b[0]` or single-child `self.b = B()` don't reproduce, even with FSDP2. - **`nn.Embedding` in the unwrapped root** — replacing with a `nn.Linear` on a float input eliminates the alias. - **`_sl` chunk-per-rank on the int input** — no chunk → no alias output. - **Aliased misaligned int64 slices** — the actual thing being miscomputed. All 10 ablations that reproduced the bug are fixed by this patch. None regressed the ones that didn't reproduce. ## Supersedes These two prior drafts targeting the WRONG mechanism are being kept for context but should be closed once this lands (both already updated with WIP status + a comment pointing here): - pytorch#190938 — dynamo SKIP-recursive workaround. Only sidesteps the symptom by forcing more of the frame to eager. - pytorch#190939 — FSDP2 idempotency guard. Verified no-op: `post_backward` fires exactly once in the buggy run; there is no double-reduce. Related bugs in the same `gen_alias_from_base` / `as_strided`-fallback family (no umbrella issue exists yet): pytorch#153041, pytorch#163328, pytorch#161937, pytorch#103650. (An earlier revision of this description cited pytorch#142358 as an umbrella; that issue is unrelated.) ## Test plan - [x] Non-distributed regression tests added in `test/inductor/test_alignment.py` (see section above); verified fail-before / pass-after against this fix: ``` python -m pytest test/inductor/test_alignment.py -k "alias_of_misaligned" -q ``` - [x] Full `test/functorch/test_aotdispatch.py`: 2852 passed, 1 failed — the failure (`TestPartitioning::test_disable_functionalization_ignores_effect_token_metadata`) is preexisting and reproduces with this fix reverted. The 8 alias-test failures from the first revision are fixed by the `_cdata` comparison: ``` python -m pytest test/functorch/test_aotdispatch.py -q python -m pytest test/functorch/test_aotdispatch.py -k alias -q # 142 passed ``` - [x] Full `test/inductor/test_alignment.py` (28 passed) and `test/functorch/test_codegen_output_alias.py` + `test_compile_to_python.py` (70 passed). - [x] Cudagraphs interaction (`mode="reduce-overhead"`): alias output lands on the user tensor at the correct offset. - [x] The distributed 2-GPU repro above (loss/grads match eager). Pull Request resolved: pytorch#191002 Approved by: https://github.com/ezyang Co-authored-by: Edward Z. Yang via mergedog <ezyang@meta.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

ethanwee1 and others added 18 commits November 21, 2025 10:09

Update install_rocm.sh

9063aed

Create therock-nightly.yml

e84c36e

Update therock-nightly.yml

668fd91

Update install_rocm.sh

bd618cd

Add TheRock nightly CI workflow for PyTorch ROCm builds

0af20bb

updates

0caf833

Delete PR_TITLE_AND_DESCRIPTION.txt

cd63b97

Delete THEROCK_NIGHTLY_PR_SUMMARY.md

e55d5a3

test

9de2742

test

c298544

test

f8d4e44

test

2040701

test

9d1cb6b

test

4278dab

Update Dockerfile

d4f11f3

Update install_rocm.sh

24b27cb

Update Dockerfile

6aeb881

Update Dockerfile

eb23cc2

ethanwee1 force-pushed the main branch from 25378ef to ce296ab Compare March 22, 2026 21:56

ethanwee1 force-pushed the main branch from 0843c3e to d416615 Compare April 23, 2026 17:08

ethanwee1 requested a review from jithunnair-amd as a code owner April 23, 2026 17:08

ethanwee1 force-pushed the main branch from d416615 to a6a3e83 Compare April 23, 2026 17:12

ethanwee1 force-pushed the main branch from fa730fd to 0168c92 Compare July 7, 2026 16:35

ethanwee1 force-pushed the main branch from 8748bc0 to fad66b2 Compare July 7, 2026 17:28

Navigation Menu

Add therock - #2

Add therock#2
ethanwee1 wants to merge 18 commits into
mainfrom
add_therock

ethanwee1 commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ethanwee1 commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant