deepmd: preload the mpich wheel's libfabric before importing deepmd - #226
Merged
Conversation
deepmd-kit's PyTorch backend preloads libmpi.so.12 from the mpich wheel at
import. That libmpi needs the libfabric bundled under lib/mpich/ (symbol
version FABRIC_1.9). On Cray systems LD_LIBRARY_PATH carries the system
libfabric, which outranks the wheel's RUNPATH and lacks that version, so
every setup() died on Delta with:
OSError: /opt/cray/libfabric/2.3.1/lib64/libfabric.so.1: version
`FABRIC_1.9' not found (required by .../libmpi.so.12)
Load the bundled libfabric by absolute path (RTLD_GLOBAL) before the first
deepmd import; the dynamic linker then reuses it when libmpi asks for
libfabric.so.1. No-op when mpich ships none.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
First Delta sync of the
deepmdenv (#225, 2026-09-04): the build succeeded, then every download failed withCause
deepmd.ptimportsdeepmd/pt/cxx_op.py, which on cibuildwheel builds callsload_mpi_library(): actypes.CDLLof thempichwheel'slibmpi.so.12(the custom-op library links MPI for the LAMMPS plugin's border op). That libmpi is built against the libfabric the wheel bundles inlib/mpich/libfabric.so.1, reached via RUNPATH. On Cray systems the module environment puts/opt/cray/libfabric/<ver>/lib64onLD_LIBRARY_PATH, which the dynamic linker consults before RUNPATH, so the Cray build is picked and it does not exportFABRIC_1.9. Rootstock's ownLD_LIBRARY_PATHprepend covers<venv>/lib(where libmpi is) but not<venv>/lib/mpich.Fix
setup()/setup_from_path()now locate the bundled libfabric throughimportlib.metadata.files("mpich")(same lookup deepmd uses for libmpi, with asys.prefix/lib/mpichfallback) andctypes.CDLL(..., RTLD_GLOBAL)it before the first deepmd import. A library already loaded under the sonamelibfabric.so.1is reused when libmpi asks for it, so the Cray copy is never opened. No-op when mpich ships no libfabric. Not Delta-specific: Perlmutter, Frontier, Polaris and Aurora all run Cray PE.Documented in the env docstring as a technical constraint. Two tests added (preload path + RTLD_GLOBAL; skip when absent).
uv run pytest tests/sample_configs: 154 passed.🤖 Generated with Claude Code