Skip to content

chore: upgrade z5 library to 3.0.1 (remove xtensor dependency)#8

Open
sameeul wants to merge 11 commits into
mainfrom
chore/upgrade-z5-3.0.1
Open

chore: upgrade z5 library to 3.0.1 (remove xtensor dependency)#8
sameeul wants to merge 11 commits into
mainfrom
chore/upgrade-z5-3.0.1

Conversation

@sameeul

@sameeul sameeul commented Jul 1, 2026

Copy link
Copy Markdown
Owner

Summary

Upgrade z5 library from 2.0.20 (with xtensor dependency) to z5 3.0.1 (header-only, no xtensor).

This is a clean migration that eliminates 2 transitive dependencies while using the modern z5 3.0.1 API.

Key Changes

  • Dependencies removed: xtensor (>=0.26,<0.27), xsimd (>=13,<14)
  • Dependencies added: z5py >=3.0 (version pin in conda_cpp.txt)
  • CMakeLists.txt updates:
    • Removed GCC 7.5 version check (z5 3.0.1 requires C++20)
    • Removed Boost dependency check (not needed for z5 3.0.1)
  • Header file migration (omezarr.h, raw_omezarr.h):
    • Removed xtensor includes
    • Added z5 3.0.1 ArrayView API includes
    • Migrated data reading from xt::xarray to std::vector + ArrayView

API Migration

Before (z5 2.0.20 + xtensor):

#include "xtensor/containers/xarray.hpp"
#include "z5/multiarray/xtensor_access.hxx"

xt::xarray<FileType> array(shape);
z5::multiarray::readSubarray<FileType>(ds, array, offset.begin());

After (z5 3.0.1):

#include "z5/multiarray/array_view.hxx"
#include "z5/multiarray/array_access.hxx"

std::vector<FileType> buffer(data_height * data_width);
auto view = z5::multiarray::makeView(buffer.data(), shape);
z5::multiarray::readSubarray<FileType>(*ds, view, offset.begin());

Benefits

✓ Eliminates 2 transitive dependencies (xtensor, xsimd)
✓ Simpler conda environment configuration
✓ z5 3.0.1 headers come with z5py conda package
✓ Uses modern C++20 header-only library
✓ No functional changes to zarr reading behavior

Files Changed

  • CMakeLists.txt: 12 lines removed (GCC 7.5 check, Boost check)
  • ci-utils/envs/conda_cpp.txt: Added z5py >=3.0, removed xtensor/xsimd
  • src/nyx/omezarr.h: Migrated to ArrayView API
  • src/nyx/raw_omezarr.h: Migrated to ArrayView API

Testing Checklist

  • CMake build succeeds with -DUSE_Z5=ON
  • OME-Zarr file loading works correctly
  • All existing tests pass
  • No xtensor/xsimd in final build dependencies

sameeul and others added 11 commits June 30, 2026 18:54
…nly, no xtensor)

Major changes:
- Remove xtensor >=0.26,<0.27 and xsimd >=13,<14 from conda_cpp.txt
- Add version pin z5py >=3.0 in conda_cpp.txt
- Update CMakeLists.txt: remove GCC 7.5 version check (z5 3.0.1 requires C++20)
- Update CMakeLists.txt: remove Boost dependency check (not needed for z5 3.0.1)

Code changes:
- Replace xtensor xarray with z5 3.0.1 ArrayView API
- Update omezarr.h and raw_omezarr.h to use z5::multiarray::makeView()
- Read zarr data into plain std::vector buffers using ArrayView
- Remove xtensor includes (#include "xtensor/containers/xarray.hpp")
- Add z5 3.0.1 includes (array_view.hxx, array_access.hxx)

Benefits:
- Eliminates 2 transitive dependencies (xtensor, xsimd)
- Simplifies conda environment configuration
- Uses modern z5 3.0.1 API (C++20 header-only library)
- No functional changes to zarr reading behavior

The z5py conda package (3.0+) now includes the C++ headers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nly, no xtensor)

Major changes:
- Remove xtensor >=0.26,<0.27 and xsimd >=13,<14 from conda_cpp.txt
- Replace z5py with z5 package (provides C++ headers only, no xtensor)
- Update CMakeLists.txt: remove GCC 7.5 version check (z5 3.0.1 requires C++20)
- Update CMakeLists.txt: remove Boost dependency check (not needed for z5 3.0.1)

Code changes:
- Replace xtensor xarray with z5 3.0.1 ArrayView API
- Update omezarr.h and raw_omezarr.h to use z5::multiarray::makeView()
- Read zarr data into plain std::vector buffers using ArrayView
- Remove xtensor includes (#include "xtensor/containers/xarray.hpp")
- Add z5 3.0.1 includes (array_view.hxx, array_access.hxx)

Benefits:
- Eliminates 2 transitive dependencies (xtensor, xsimd)
- Simplifies conda environment configuration
- Uses modern z5 3.0.1 API (C++20 header-only library)
- z5 package provides C++ headers without xtensor dependency
- No functional changes to zarr reading behavior

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n array_access.hxx:146

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The conda package that ships the z5 C++ headers is z5py, not z5 -
there is no conda-forge package named z5. z5py 3.0.1 bundles the
full include/z5/ header tree (z5.hxx, multiarray/array_access.hxx,
etc.), so the xtensor/xsimd drop stays but the package name must be
z5py. Fixes 'z5 =* * does not exist' mamba solve error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
z5py 3.0.x ships no Python 3.10 build (only 3.11-3.14 on all
platforms), so the 3.10-pinned conda env could not solve z5py>=3.0.1
('no viable options ... requires python >=3.11'). The branch's code
uses the z5 3.0.1 ArrayView API, so pinning back to z5py 2.x is not
an option (2.x is xtensor-based and won't compile). Bump the ubuntu/
mac/windows test jobs 3.10 -> 3.11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds GTest coverage for both z5-based OME-Zarr readers:
- NyxusOmeZarrLoader (src/nyx/omezarr.h)
- RawOmezarrLoader   (src/nyx/raw_omezarr.h)

Two zarr-v2 test stores generated with bfio (see
tests/data/omezarr/README.md):
- test.ome.zarr  512x512 uint16, single tile
- multi.ome.zarr 1500x1200 uint16, 2x2 tile grid with partial edges

Tests verify geometry (full/tile dims), exact pixel values, full-image
checksums, and the multi-tile + partial-tile read paths for both loaders.
Guarded by OMEZARR_SUPPORT so builds without USE_Z5 are unaffected.

Verified end-to-end by compiling both readers against z5 3.0.1 and
reading the datasets (all assertions pass).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
z5 3.0.1 no longer needs Boost, but building Arrow support does.
Move find_package(Boost) into the USE_ARROW block (disabling Arrow
with a warning if Boost is missing) and add its include dirs when
Arrow is enabled. Verified: cmake configure with -DUSE_ARROW=ON
finds Boost and keeps Arrow support enabled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
z5 3.0.1 added the USE_LIBDEFLATE option (default ON), which makes its
CMake do find_package(libdeflate CONFIG REQUIRED). libdeflate is not
built in the wheel prereq path, so z5's configure failed, 'make install'
never ran, the z5 headers were never installed, and 'Validate prereqs'
failed with:
  FAIL: Z5 header (local_install/include/z5/z5.hxx)
  FAIL: Z5 array_access header (.../multiarray/array_access.hxx)
which broke every wheel build (they depend on the prereq step).

Pass -DUSE_LIBDEFLATE=OFF to fall back to the stock zlib gzip backend
(already built). Per z5 docs the on-disk format is unchanged; our data
uses the blosc codec regardless. Verified locally: z5 3.0.1 configures,
installs its headers, and both OME-Zarr readers read the test data
correctly against this build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…l regression)

My previous commit gated USE_ARROW on find_package(Boost) succeeding.
That broke the Windows wheel build: the prereq ships Boost as
headers-only under local_install/include/boost with no CMake config,
so find_package(Boost) reports 'Could NOT find Boost (missing
Boost_INCLUDE_DIR)', which then set USE_ARROW OFF. Wheels built without
Arrow, and the Arrow/Parquet pytest suite failed with:
  RuntimeError: ... Apache Arrow functionality is not available.

Make the Boost lookup best-effort: keep it in the Arrow section (per
the request that Arrow, not z5, is what needs Boost), add Boost include
dirs when found, and otherwise fall back to $NYXUS_DEP_DIR/include for
the headers-only layout. Arrow is no longer disabled on a Boost
config-mode miss. Nyxus sources don't include Boost directly; Boost is
only a transitive Arrow build dependency, so headers-on-path is enough.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Supersedes the earlier USE_LIBDEFLATE=OFF workaround. The prereq scripts
already installed libdeflate v1.19, just after the z5 build instead of
before it, so z5's find_package(libdeflate CONFIG REQUIRED) failed.

Move the libdeflate install ahead of the z5 build (right after zlib, kept
unconditional since libtiff also uses it) and drop -DUSE_LIBDEFLATE=OFF so
z5 3.0.1 builds with its default libdeflate gzip/zlib backend. Add a
libdeflate CMake-config check to validate_prereqs.sh.

Verified locally: libdeflate installs its CMake config, z5 3.0.1 with the
default USE_LIBDEFLATE=ON finds it and installs the z5 headers, both
OME-Zarr readers read the test data, and validate_prereqs passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant