fix: upgrade Docker build to CUDA 12.6 and Python 3.11#6
Open
sameeul wants to merge 6 commits into
Open
Conversation
Upgrade the OME-Zarr readers from z5 2.0.20 (xtensor-based) to the header-only z5 3.0.1 ArrayView API, removing the xtensor and xsimd dependencies entirely. - omezarr.h / raw_omezarr.h: replace xt::xarray + xtensor_access with z5::multiarray::makeView + array_access.hxx; read into plain std::vector buffers. No change to on-disk read behavior. - CMakeLists.txt: drop the GCC 7.5 xtensor guard and the Boost find_package from the Z5 detection block (z5 3.0.1 needs neither). - conda_cpp.txt / conda_cpp_mac.txt / environment.yml: pin z5py >=3.0.1 (the conda package that ships the z5 C++ headers) and remove xtensor / xsimd. BREAKING CHANGE: OME-Zarr support now requires z5 3.0.1 and C++20; xtensor/xsimd are no longer used.
Update the wheel-build prereq scripts for the z5 3.0.1 upgrade: - Remove the xtl / xtensor / xsimd source builds (no longer needed). - Bump z5 from 2.0.20 to 3.0.1. - Install libdeflate before the z5 build so z5 3.0.1's find_package(libdeflate CONFIG REQUIRED) resolves (libdeflate was already built later in the script; it is moved up and kept unconditional since libtiff also depends on it). - validate_prereqs.sh: drop the xtensor header check, add checks for z5 multiarray/array_access.hxx and the libdeflate CMake config.
z5py 3.0.x ships no Python 3.10 build (only 3.11-3.14 on all platforms), so the 3.10-pinned conda environment could not solve z5py >=3.0.1. Bump the ubuntu, mac and windows test jobs to 3.11.
z5 3.0.1 no longer needs Boost, but Apache Arrow does. Move the Boost lookup from the Z5 block to the Arrow block. Keep it best-effort: find_package(Boost QUIET) adds the include dirs when available, and otherwise the build falls back to Boost headers under $NYXUS_DEP_DIR/include (some prereq layouts ship Boost headers-only with no CMake config). Arrow is not disabled on a Boost config-mode miss, so headers-only Boost still builds.
Add GTest coverage for both z5-based OME-Zarr readers (NyxusOmeZarrLoader and RawOmezarrLoader): geometry, exact pixel values, full-image checksums, and multi-tile / partial-tile reads. Ships two small bfio-generated zarr-v2 datasets under tests/data/omezarr (512x512 single-tile and 1500x1200 2x2 tile grid with partial edges); see the datasets' README for regeneration. Tests are guarded by OMEZARR_SUPPORT so non-Z5 builds are unaffected.
- Replace cudatoolkit=11.5 conda packages with cuda-version=12.6, cuda-nvcc=12.6, cuda-cudart-dev=12.6, libcufft-dev (conda-forge CUDA 12 package names that include nvcc) - Add -DCMAKE_CUDA_COMPILER=$CONDA_PREFIX/bin/nvcc so CMake finds the conda nvcc without relying on check_language(CUDA) discovery - Bump PYTHON=3.9 -> 3.11 in build_conda.sh (3.9 is EOL) - Update Dockerfile base image from nvidia/cuda:11.5.2-runtime-ubuntu20.04 to nvidia/cuda:12.6.0-runtime-ubuntu22.04 - Update hardcoded nyxus-3.9 env path in publish_docker.yml to nyxus-3.11 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sameeul
force-pushed
the
fix/docker-cuda12-python311
branch
from
91b9f09 to
7ec899d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades the GPU Docker build to CUDA 12.6 and Python 3.11:
nvidia/cuda:11.5.2-runtime-ubuntu20.04→nvidia/cuda:12.6.0-runtime-ubuntu22.04conda_linux_gpu.txt: replacecudatoolkit=11.5/cudatoolkit-dev=11.5with the CUDA 12.6 conda-forge packages that actually shipnvcc(cuda-version=12.6,cuda-nvcc=12.6,cuda-cudart-dev=12.6,libcufft-dev)build_conda.sh: add-DCMAKE_CUDA_COMPILER=$CONDA_PREFIX/bin/nvccso CMake doesn't rely oncheck_language(CUDA)discovery, and bumpPYTHON=3.9→3.11(3.9 is EOL)publish_docker.yml: update the hardcodednyxus-3.9env path tonyxus-3.11Root cause (the CUDA fix)
cudatoolkit-dev=11.5.0installs headers/libs but doesn't placenvccwhere CMake'scheck_language(CUDA)looks, so CMake reportedCMAKE_CUDA_COMPILER=NOTFOUNDand hit the fatal error inCMakeLists.txt. The CUDA 12.6 conda-forge packages shipnvcc, and passing-DCMAKE_CUDA_COMPILERexplicitly makes discovery deterministic.Rebased onto the z5 3.0.1 upgrade
This branch previously carried two extra commits that worked around the old z5 2.0.20 + xtensor stack in the Docker build:
z5/multiarray/xtensor_access.hxxinstead ofz5/z5.hxx"Both are moot now that the z5 3.0.1 upgrade is merged into
main: CMake looks forz5/z5.hxxagain, and the z5 C++ headers come from thez5py >=3.0.1conda package inconda_cpp.txt(no manual header install needed). The branch has been rebased onto the currentmainand those two commits dropped, leaving only the Docker/CUDA/Python change.Note the Python 3.11 bump is now also required for the GPU build to solve, not just a nicety: z5py 3.0.x has no Python 3.10/3.9 build (3.11+ only), and
build_conda.shpulls z5 viaconda_cpp.txt(z5py >=3.0.1).Test plan
Publish Docker imageworkflow on this branch and confirm the "Run Build Script" step passes (CUDA build + z5 3.0.1 via z5py on Python 3.11).🤖 Generated with Claude Code