diff --git a/.github/workflows/hypothesis.yaml b/.github/workflows/hypothesis.yaml index dd49578dd0..f463397c85 100644 --- a/.github/workflows/hypothesis.yaml +++ b/.github/workflows/hypothesis.yaml @@ -25,6 +25,9 @@ jobs: hypothesis: name: Slow Hypothesis Tests + permissions: + contents: read + issues: write environment: name: codecov-upload deployment: false @@ -83,6 +86,7 @@ jobs: id: status env: HATCH_ENV: test.py${{ matrix.python-version }}-${{ matrix.dependency-set }} + PYTEST_ADDOPTS: "--report-log=output-${{ matrix.python-version }}-log.jsonl" run: | echo "Using Hypothesis profile: $HYPOTHESIS_PROFILE" hatch env run --env "$HATCH_ENV" run-hypothesis @@ -113,4 +117,4 @@ jobs: with: log-path: output-${{ matrix.python-version }}-log.jsonl issue-title: "Nightly Hypothesis tests failed" - issue-label: "topic-hypothesis" + issue-label: "automated issue" diff --git a/changes/4149.doc.md b/changes/4149.doc.md new file mode 100644 index 0000000000..8a473acac9 --- /dev/null +++ b/changes/4149.doc.md @@ -0,0 +1 @@ +Added a Roadmap page to the documentation outlining future plans and intended changes to the library. diff --git a/changes/4260.bugfix.md b/changes/4260.bugfix.md new file mode 100644 index 0000000000..b703c47a35 --- /dev/null +++ b/changes/4260.bugfix.md @@ -0,0 +1 @@ +The `cast_value` codec now requires `cast-value-rs>=0.4.2`. Earlier versions of that backend silently corrupted data when handed an array that was not row-major — the layout the `transpose` codec produces — so a `cast_value` codec next to a `transpose` codec would either write transposed values with no error or fail with `ValueError: Input array must be contiguous`. The minimum version is enforced at runtime as well as in the package metadata, so an environment that already has an older `cast-value-rs` installed now raises `ImportError` when the codec is used, instead of corrupting data. diff --git a/changes/4265.bugfix.md b/changes/4265.bugfix.md new file mode 100644 index 0000000000..6daf0fc7d0 --- /dev/null +++ b/changes/4265.bugfix.md @@ -0,0 +1,13 @@ +Accept [universal-pathlib](https://github.com/fsspec/universal_pathlib) `UPath` objects wherever +zarr accepts a `StoreLike` value. A remote `UPath` now creates an `FsspecStore` using the +filesystem and storage options the `UPath` already carries, and a local `UPath` creates a +`LocalStore`, so that `UPath('/data')` and `Path('/data')` behave the same. + +Previously this worked only by accident: in universal-pathlib < 0.3 every `UPath` subclassed +`pathlib.Path` and implemented `__fspath__`, so remote paths were either converted to a URI string +by the caller or wrapped in a `LocalStore` that happened to dispatch through fsspec. Since +universal-pathlib 0.3 remote paths do neither, and passing one raised +`TypeError: Unsupported type for store_like`. + +`FsspecStore.from_upath` also now converts the `UPath`'s filesystem to async mode, instead of +raising `TypeError` for synchronous filesystems and warning for sync-mode instances of async ones. diff --git a/changes/4279.bugfix.md b/changes/4279.bugfix.md new file mode 100644 index 0000000000..7d99a49ccf --- /dev/null +++ b/changes/4279.bugfix.md @@ -0,0 +1 @@ +A `scale_offset` codec configured with a string-valued zero scale is now rejected. `scale` accepts strings, and no string is ever equal to `0`, so `"0"`, `"0.0"` and the hex form `"0x0000000000000000"` skipped the "scale must be non-zero" check that the numeric `0` triggers. On float data types the array was created, every chunk was written as zero and read back as `nan` with no error, and the zero scale was persisted to the metadata so reopening the store reproduced it; on integer data types the codec raised `ZeroDivisionError` instead of `ValueError`. The check now runs on the parsed scalar rather than the value as supplied. diff --git a/changes/4288.bugfix.md b/changes/4288.bugfix.md new file mode 100644 index 0000000000..4080c2b44c --- /dev/null +++ b/changes/4288.bugfix.md @@ -0,0 +1,3 @@ +`zarr.from_array` now defaults to the fill value and the attributes of the source array. Previously both were silently discarded: the array was created with the data type's default scalar and no attributes. + +An explicit `fill_value=None` now selects the data type's default scalar (Zarr format 3) or a null fill value (Zarr format 2), consistently with `create_array`, and an empty `attributes` dict creates the array with no attributes. diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000000..4b5dbc4599 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,189 @@ +# Roadmap + +This page describes where Zarr-Python is headed: the goals for the next major +cycle of work, the changes we intend to make, and how those changes will be +released. It is a living document; discussion and counter-proposals are welcome +on the +[Zarr-Python issue tracker](https://github.com/zarr-developers/zarr-python/issues). + +*The history of this roadmap, including the detailed technical proposals it +was distilled from, can be traced in the +[zarr-python-planning](https://github.com/zarr-developers/zarr-python-planning) +repository.* + +!!! note + + This roadmap reflects the current thinking of the core developers. It is a + statement of direction, not a schedule. We don't know how long these changes + will take, only that we are committed to moving the project in the direction outlined + here. + +## Where we are + +The [3.0 release](https://github.com/zarr-developers/zarr-python/releases/tag/v3.0.0) +was a total redesign of the library's internals, with three goals: full support +for the Zarr V2 and V3 storage formats, storage APIs that are ergonomic for high-latency +storage (such as cloud storage), and backwards compatibility with Zarr-Python 2.x where +possible. Those goals were largely achieved! Going by the content of issues and pull requests +submitted to the library, few users are grappling with 2.x → 3.x migration issues. Instead, we see +users asking for things like better APIs, where "better" usually means faster. + +The 3.x redesign was carried out under hard backwards-compatibility +constraints, and it inherited many structural patterns from the 2.x +implementation it replaced. The library has never had a release cycle whose +primary goal was the *shape* of the internals. The next body of work — which we +call **"v4"** — is that overdue investment. We think iterating on the internals of +the library will make it *much* easier to bring faster, more expressive APIs to Zarr-Python +users. + +## Goals + +If the 3.0 goals could be sloganized as "migrate to Zarr V3, and improve cloud +storage support", the slogan for the v4 goals is: +**"a frictionless Zarr-based Python ecosystem for chunked arrays"**. Zarr-Python +should be *foundational* for the growing number of Python packages that work +with data in the Zarr format. Concretely, that means pushing in these +directions: + +- Deliver excellent performance, out of the box, while retaining maintainability. +- Make Zarr-Python APIs ergonomic and useful for developers. +- Expand our scope to cover vital quality-of-life routines like data copying, + rechunking, and the like. +- Ease the growth of Python tools across all levels of the Zarr stack. +- Accelerate the implementation of new codecs, chunk grids, chunk key + encodings, etc. + +An important design input: [`zarrs`](https://github.com/zarrs/zarrs) (Rust) and +[TensorStore](https://github.com/google/tensorstore) (C++) are two independent +Zarr implementations that use architectural patterns we want to learn from. +We see them as complementary rather than competitive. + +!!! note + + Many of the features in this roadmap will not require breaking public 3.x APIs. We can and will + ship those features in 3.x releases; at the same time, we consider it clarifying to frame the + coherent development direction as vectored at a 4.0 milestone. + +## The Zarr stack + +Different applications need different levels of Zarr support: a convention +validator only needs to read metadata documents; a visualization tool may only +need read-only array access; other tools need everything. We think of this as a +"Zarr stack", from most abstract to most concrete: + +1. **Conventions** — application and/or domain-specific schemas built on top of Zarr (OME-NGFF, + GeoZarr, anndata-zarr, multiscales). +2. **Groups** — Zarr hierarchies, traversal, group-level attributes. +3. **Arrays** — the user-facing array object, plus indexing and slicing. +4. **Chunk decoding** — the codec pipeline. +5. **Chunk addressing** — chunk grids and key encodings that map array + coordinates to store keys. +6. **Stores** — the key-value layer. +7. **Metadata** — pure data documents describing arrays and groups. + +Today, Zarr-Python is a monolith that serves every level: a consumer who only +needs metadata handling has to install the full dependency footprint of the +whole library, and a faster chunk-decoding implementation cannot plug in +without re-implementing the layers above it. The v4 direction is to re-shape +Zarr-Python around the stack, so that each level is something you can depend +on, conform to, or replace, without buying every other level. + +We plan to "stackify" Zarr-Python by spinning core functionality out into separate Python packages, e.g. `zarr-metadata`, `zarr-indexing`, `zarr-storage`, +`zarr-codec`, `zarr-dtype`, each with narrow scope, all composed in the `zarr` package. The Rust `zarrs` library +successfully uses a structure like this, and we are keen to share the benefits of a more modular, maintainable codebase. Two of these subpackages, +[`zarr-metadata`](https://zarr.readthedocs.io/projects/zarr-metadata/en/latest/) and [`zarr-indexing`](https://zarr.readthedocs.io/projects/zarr-indexing/en/latest/), are already +published. + +## What we intend to change + +The following section details how we want to evolve the internal logic that drives Zarr-Python. + +### Foundation: swappable backends + +We propose to refactor Zarr-Python internals around a *swappable engine* — a protocol, or protocols, +that define the core routines a Zarr implementation must support. Zarr-Python becomes one user-facing +API that can be driven by multiple backends, including externally defined backends. We think this will allow users on many different platforms to get the best performance for their particular environment while retaining a familiar API. + +#### Rust bindings + +We want a Python backend (i.e., the status quo), but also a Rust-based backend, via bindings to the +[`zarrs`](https://docs.rs/zarrs/latest/zarrs/) crate. The [zarrs-python](https://zarrs-python.readthedocs.io/en/latest/) project demonstrates that +bridging `zarrs` and Zarr-Python buys a *lot* of performance in the specific case of chunk encoding. But zarrs-python is constrained today by limited +modularity in Zarr-Python internals. Refactoring our internals around swappable backends should address this limitation. + +Any Python package that interfaces with `zarrs` will need Pythonic bindings to the Rust library. So we are *very* excited about the [zarrista](https://developmentseed.org/zarrista/latest/) package, which aims to provide complete Python bindings for `zarrs`. + +#### Sync / Async partitioning + +Internally we will branch over two kinds of backends: synchronous and asynchronous. The synchronous backend is suitable for arrays and groups persisted to low-latency storage like in-memory stores or local file systems, where async scheduling is pure friction. The asynchronous backend will use Python's `async` support and will provide concurrent APIs where it helps: for arrays and groups persisted to high-latency storage. + +### Lazy indexing + +The Zarr-Python Array API was initially designed to mirror NumPy, with eager +array indexing syntax. `Array.__getitem__` performs IO eagerly and returns a NumPy array. +That was helpful to the dominant use-case at the time of its creation, but it +means deferred IO and computation currently require an external library +such as Dask. It means there is no built-in support for representing multi-step +reads as a single deferred plan. Further, it means that every chained +selection round-trips to storage independently. + +We can fix this by introducing an API for lazy indexing. Under this model, an array indexing operation +like `array[::2]` desugars to a declarative state like `(array, selection)`. Chained selections like +`array[10:100][::2]` are fused immediately, and we defer actual IO for the time when the result of +indexing is needed. [TensorStore](https://google.github.io/tensorstore/) is an excellent role model +for Zarr-Python here, and we can deliver this functionality without breaking ordinary indexing behavior. +See this [discussion](https://github.com/zarr-developers/zarr-python/discussions/1603) for more +background. + +### Data types + +First-class support for ML-specific dtypes — `bfloat16`, the `float8` +variants, packed `int4`/`uint4` — via +[`ml_dtypes`](https://github.com/jax-ml/ml_dtypes). These data types have specifications written up in `zarr-extensions`, but there's no simple to get them integrated in Zarr-Python today. + +### Device-agnostic IO + +Make Zarr-Python's IO surfaces device-agnostic rather than adding GPU support +as a bolted-on feature: stores and codecs grow APIs for writing into a +caller-provided buffer (`read_into`, `decode_into`), and the `Array` facade +returns array-like objects in the user's chosen Array API namespace. GPU +support falls out once the assumption of CPU destinations is removed, and CPU +paths get faster too, because pre-allocated output buffers eliminate per-chunk +allocation. + +### Configuration, registries, and plugins + +Move configuration from "global mutable state read implicitly" to "typed data +passed explicitly": a typed config object replacing the untyped global `donfig` +dict, array-scoped runtime config passed at open time, a registry redesign that +addresses implementations by stable identity and resolves plugin name-conflicts +deliberately, and named profiles replacing global mutators. + +### Coordinated and distributed writes + +This area is actually an unfinished aspect of the 2.x → 3.0 migration: Zarr-Python 2.x supported +synchronization logic via file-based locks, and we have not implemented equivalent functionality in +3.x. We don't have *concrete* plans for closing this gap. Re-implementing simple object-based locking, for +backends that support it, is a direct solution we should consider. But a transactional storage model, +where a sequence of basic storage operations like reading and writing could be submitted in a batch and +executed serially, with rollbacks under failure, is also quite appealing. +As with array indexing, TensorStore is the trailblazer here, and we can learn from its example. + +We can also avoid the need for synchronization mechanisms entirely with better planning. +Many users of the 2.x synchronization tooling needed to simply write values from one chunked source +to another, without worrying about chunk alignment. This can be addressed e.g. by creating a write +plan that partitions the input chunks into batches within which writes cannot race. + +## How to get involved + +- **Discuss the plans.** Comments and counter-proposals on any of the themes + above are welcome on the + [issue tracker](https://github.com/zarr-developers/zarr-python/issues) and in + the [developer chat](https://ossci.zulipchat.com/). +- **Review in-flight work.** The `IndexTransform` algebra that lazy indexing is + built on is in review at + [#3906](https://github.com/zarr-developers/zarr-python/pull/3906). +- **Weigh in as a downstream maintainer.** If your project's use of + Zarr-Python would be affected by the codec API rewrite, the stores rewrite, + or the lazy-indexing work, the planning phase is the time to surface + workloads or patterns that don't fit. diff --git a/docs/user-guide/storage.md b/docs/user-guide/storage.md index b288c9976d..a34e2e2874 100644 --- a/docs/user-guide/storage.md +++ b/docs/user-guide/storage.md @@ -90,6 +90,16 @@ print(group) - an FSSpec [FSMap object](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.FSMap), which will create an [FsspecStore](#remote-store). +- a [universal-pathlib](https://github.com/fsspec/universal_pathlib) `UPath`, which will create an + [FsspecStore](#remote-store), or a [local store](#local-store) if the `UPath` is local. Put your + storage options on the `UPath` itself; passing a separate `storage_options` argument alongside + one raises `TypeError`. + + ```python exec="false" reason="requires universal-pathlib, which is not in the docs environment" + from upath import UPath + group = zarr.open_group(UPath('s3://noaa-nwm-retro-v2-zarr-pds', anon=True), mode='r') + ``` + - a [`Store`][zarr.abc.store.Store] or [`StorePath`][zarr.storage.StorePath] - see explicit store creation below. diff --git a/lychee.toml b/lychee.toml index 54a5b49b8d..dccb3001dc 100644 --- a/lychee.toml +++ b/lychee.toml @@ -22,4 +22,6 @@ exclude = [ '^https?://(localhost|127\.0\.0\.1)(:\d+)?', # SPEC 0 page times out but is valid. '^https://scientific-python\.org/specs/spec-0000', + # Stack Overflow answers link-checker requests with 403; the tag page is valid. + '^https://stackoverflow\.com/questions/tagged/', ] diff --git a/mkdocs.yml b/mkdocs.yml index 1fde8d9fe3..4d06701a87 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -102,6 +102,7 @@ nav: - 'zarr-metadata ↪': https://zarr.readthedocs.io/projects/zarr-metadata/ - 'zarr-indexing ↪': https://zarr.readthedocs.io/projects/zarr-indexing/ - release-notes.md + - roadmap.md - contributing.md - Blog: - blog/index.md @@ -221,7 +222,6 @@ plugins: 'search.html.md': 'index.md' 'tutorial.md': 'user-guide/installation.md' 'getting-started.md': 'quick-start.md' - 'roadmap.md': 'https://zarr.readthedocs.io/en/v3.0.8/developers/roadmap.html' 'installation.md': 'user-guide/installation.md' 'release.md': 'release-notes.md' 'about.html.md': 'index.md' diff --git a/packages/zarr-http-server/changes/4291.doc.md b/packages/zarr-http-server/changes/4291.doc.md new file mode 100644 index 0000000000..18e8fb71d5 --- /dev/null +++ b/packages/zarr-http-server/changes/4291.doc.md @@ -0,0 +1 @@ +Updated the MDN CORS link in the user guide to its current URL. diff --git a/packages/zarr-http-server/docs/guide.md b/packages/zarr-http-server/docs/guide.md index d52d5b882d..5a2658d86b 100644 --- a/packages/zarr-http-server/docs/guide.md +++ b/packages/zarr-http-server/docs/guide.md @@ -215,7 +215,7 @@ bind has no URL to report. Both app builders accept a [`CorsOptions`][zarr_http_server.CorsOptions] parameter to enable -[CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) middleware for +[CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS) middleware for browser-based clients: ```python diff --git a/packages/zarr-indexing/changes/4291.doc.md b/packages/zarr-indexing/changes/4291.doc.md new file mode 100644 index 0000000000..a1e69fad0b --- /dev/null +++ b/packages/zarr-indexing/changes/4291.doc.md @@ -0,0 +1,2 @@ +Rewrote the guide's prev/next navigation links as markdown so mkdocs +validates them at build time; the rendered pages are unchanged. diff --git a/packages/zarr-indexing/docs/guide/index.md b/packages/zarr-indexing/docs/guide/index.md index 780cfd7e56..42e47530fb 100644 --- a/packages/zarr-indexing/docs/guide/index.md +++ b/packages/zarr-indexing/docs/guide/index.md @@ -437,8 +437,8 @@ correspondence between every request position and its chunk-local source cell. --- -