Add IVCSCArray/IVCSRArray with a normalized view (#5) - #6
Open
aarmey wants to merge 1 commit into
Open
Conversation
…lized view Implements GH issue #5: an alternative to rapid_load that keeps IVCSR/IVCSC data byte-packed in memory instead of decoding it. IVCSCArray/IVCSRArray support indexing/subsetting (staying packed) and per-axis read sums (major-axis sums skip decoding entirely); anything else (matmul, elementwise arithmetic, ufuncs) raises RuntimeError. VCSCAnnData now accepts either array family for X/raw_X and gained real (eager) indexing/view support. load_packed() loads an on-disk IVCSR/IVCSC .h5ad straight into this packed form, with IO round-trip support registered for both storage formats. Also adds IVCSCArrayNormalized/IVCSRArrayNormalized: a read-depth-normalized, log-transformed, mean-centered view over these arrays that never materializes the (necessarily dense, once centered) result. Only three O(n_rows)/O(n_cols) statistics are precomputed, via parallel numba kernels specialized per storage format; toarray() and elementwise access apply the transform on demand from those stats plus the raw packed data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PswzuSVAmYkh5yNbSMHAnz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #5: an alternative to
rapid_loadwhere IVCSR/IVCSC data loaded into memory stays in its packed form instead of being decoded/normalized.IVCSCArray/IVCSRArray(_ivcs.py): keep the minor-axis indices byte-packed (delta+varint, as on disk) in memory. Support indexing/subsetting (result stays packed) and per-axis read sums (sum(axis=0|1)— major-axis sums never decodeindices, minor-axis sums decode once and cache). Matrix products, elementwise arithmetic, and ufuncs raiseRuntimeErrorfor now.VCSCAnnDatanow accepts either the existing VCS types or the new IVCS types forX/raw_X, and gained a real (eager-copy)__getitem__supporting the usual label/int/slice/boolean indexing.load_packed(): reads an on-disk IVCSR/IVCSC.h5adstraight into aVCSCAnnDatawithXas a packedIVCSCArray/IVCSRArray— no filtering, no normalization, no decode. IO round-trip (read + write) is also registered for the new types via the standardwrite_h5ad/write_zarr/read_h5ad/read_zarrpath.IVCSCArrayNormalized/IVCSRArrayNormalized(_ivcs_norm.py, via.normalized()): a read-depth-normalized, log-transformed, mean-centered view that never materializes the (necessarily dense, once centered) result. Only three small per-row/per-column statistics are precomputed (via parallel numba kernels specialized per storage format);.toarray()and elementwise access apply the transform on the fly from those stats plus the raw packed data.Test plan
pytest -q— 891 passed, 38 skippedruff check src tests— cleanty check src— cleanIVCSCArrayNormalized/IVCSRArrayNormalizedagainst a from-scratch numpy reference implementation of the depth-normalize/log-transform/mean-center pipeline, for both storage formats and edge cases (all-zero matrix, zero-count rows, boolean/int slicing).🤖 Generated with Claude Code
https://claude.ai/code/session_01PswzuSVAmYkh5yNbSMHAnz