macOS: give the shared libraries @loader_path install names (fixes cross-linking with pyscf) - #3
Open
smvinko wants to merge 1 commit into
Open
Conversation
On macOS every library here is stamped with an @rpath/<name>.dylib install name (CMAKE_MACOSX_RPATH ON), and pyscf ships libcgto, libnp_helper, libcvhf and libpbc under exactly those names. dyld satisfies an @rpath dependency from any already-loaded image whose install name matches -- it does not consult the referring library's own LC_RPATH first -- so importing pyscf and dqclibs into one process cross-links the two stacks according to import order. Measured on macOS 15 (arm64), python 3.14, pyscf 2.13.1: import pyscf; import dqclibs; dqclibs.CPBC() OSError: dlopen(.../dqclibs/libpbc.dylib): Symbol not found: _GTO_aopair_lazy_contract (Expected in: .../pyscf/lib/libcgto.dylib) dqclibs' own libcgto exports that symbol; pyscf's does not. import dqclibs; dqclibs.CGTO(); import pyscf; dqclibs.CPBC() loads, then SIGSEGV in PBC_ft_latsum_drv, and periodic integrals silently return WRONG NUMBERS rather than crashing. Either way, a periodic KS energy changed merely because an earlier calculation in the same process had imported pyscf. pyscf is immune because it links its own siblings by @loader_path. This does the same for dqclibs, in two parts: * libs/CMakeLists.txt -- CMAKE_INSTALL_NAME_DIR "@loader_path" covers libcgto, libcvhf, libpbc, libnp_helper and libsymm. * libs/libcint/CMakeLists.txt -- libcint needs a POST_BUILD install_name_tool -id instead. It is built through ExternalProject_Add straight into CMAKE_LIBRARY_OUTPUT_DIRECTORY with no install() step, so CMAKE_INSTALL_NAME_DIR never applies to it; and setting MACOSX_RPATH OFF (tried both as a -D from the parent and as a target property) makes cmake fall back to an absolute build-tree path, which is worse than the @rpath it replaces. Rewriting the id after the link is the only form that survives both. Note also that libcint.dylib, libcint.4.dylib and libcint.4.0.7.dylib all share the single id @rpath/libcint.4.dylib, so they collide with each other as well as with pyscf. Verified from a clean build (rm -rf dqclibs/deps first -- ExternalProject_Add installs libcint into ${PROJECT_SOURCE_DIR}/deps, inside the SOURCE tree, so a plain rebuild silently reuses a stale libcint): * zero @rpath references across all eight dylibs, ids and dependencies * dqclibs.CPBC() loads in both import orders * the downstream DQC test suite goes from 28 periodic tests skipped to 28 passing, and a periodic KS energy no longer depends on whether an earlier test imported pyscf No effect on Linux or Windows: both hunks are inside `if (APPLE)`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On macOS every library in
dqclibsis stamped with an@rpath/<name>.dylibinstall name (
CMAKE_MACOSX_RPATH ON), and pyscf shipslibcgto,libnp_helper,libcvhfandlibpbcunder exactly those names.dyld satisfies an
@rpathdependency from any already-loaded image whoseinstall name matches — it does not consult the referring library's own
LC_RPATHfirst. So importing pyscf and dqclibs into one process cross-linksthe two stacks according to import order.
Measured on macOS 15 (arm64), Python 3.14, pyscf 2.13.1:
dqclibs' ownlibcgtoexports that symbol; pyscf's does not.The silent-wrong-numbers mode is the reason this is worth fixing rather than
documenting: in the downstream DQC test suite, a periodic KS energy changed
merely because an earlier test in the same process had imported pyscf.
pyscf itself is immune precisely because it links its own siblings by
@loader_path.Fix
Two hunks, both inside
if (APPLE)— no effect on Linux or Windows.libs/CMakeLists.txt—set(CMAKE_INSTALL_NAME_DIR "@loader_path").Covers
libcgto,libcvhf,libpbc,libnp_helper,libsymm.libs/libcint/CMakeLists.txt— aPOST_BUILD install_name_tool -id.libcint needs a different form: it is built through
ExternalProject_Addstraight into
CMAKE_LIBRARY_OUTPUT_DIRECTORYwith noinstall()step, soCMAKE_INSTALL_NAME_DIRnever applies to it. SettingMACOSX_RPATH OFFinstead (tried both as a
-Dfrom the parent and as a target property)makes CMake fall back to an absolute build-tree path, which is worse
than the
@rpathit replaces. Rewriting the id after the link is the onlyform that survives both.
Worth noting separately:
libcint.dylib,libcint.4.dylibandlibcint.4.0.7.dyliball carry the single id@rpath/libcint.4.dylib, sothey collide with each other as well as with pyscf.
Verification
From a clean build:
@rpathreferences across all eight dylibs, ids and dependenciesdqclibs.CPBC()loads in both import ordersperiodic KS energy no longer depends on whether an earlier test imported
pyscf
Note for review
The
codesign -f -s - ... || trueafter the id rewrite is deliberatelynon-fatal (macOS invalidates the signature when the load commands change), but
it is a no-op where
codesignis absent — happy to guard it or drop it if youwould rather not shell out.