V1 prep

[parashardhapola]

Scarf 1.0 modernizes storage and cloud I/O, improves feature-wise performance, introduces a new plotting API, adds batch-aware reference mapping, and reorganizes the codebase into focused domain packages.

Existing Zarr v2 stores remain supported. No conversion is required.

See the migration notes, tutorials, and API reference for complete examples.

---

Upgrade checklist

1. Create a Python 3.12+ environment.
2. Install Scarf with uv.
3. Open an existing store and smoke-test the workflow.
4. Replace Dask-specific operations with ChunkedArray.
5. Replace legacy plotting calls with scarf.plotting.
6. Update imports from retired internal modules.
7. Rebuild WNN graphs if used.
8. Build a mapping reference for harmonized atlas mapping.
9. Optionally repack stores or add countsT for faster feature-wise analysis.

uv venv --python 3.12
source .venv/bin/activate
uv pip install 'scarf[extra]'

---

Compatibility and breaking changes

Existing stores

• Zarr v2 stores remain readable and writable.
• Stores are not converted or rewritten when opened.
• New stores use Zarr v3, sharding, and feature-major countsT.
• Existing stores continue to use counts unless explicitly upgraded.
• Existing marker trees and supported legacy ANN indexes remain readable.

Removed or replaced APIs

Previous API	Replacement
dask.array.Array	scarf.matrix.ChunkedArray
scarf.plots, DataStore.plot_*	scarf.plotting
NaboH5Reader, NaboH5ToZarr	Other supported readers and writers
to_polars_dataframe	to_pandas_dataframe
Prenormed marker arguments	Standard marker search
scarf.harmony	scarf.embeddings.harmony
scarf.chunked	scarf.matrix
scarf.mapping_reference	scarf.mapping

Documented top-level imports and DataStore methods remain the supported compatibility surface. Old private implementation modules no longer have forwarding shims. See the internal import migration table for replacements.

Deprecations retained through Scarf 1.x

• ZarrMerge -> AssayMerge
• metric_integration -> metric_label_concordance
• exclude_missing=True -> missing_feature_policy="intersection"
• ref_mu, ref_sigma, and run_coral mapping arguments

These emit DeprecationWarning and may be removed in Scarf 2.0.

Dask migration

# Before
result = ds.RNA.rawData.sum(axis=0).compute()

# Scarf 1.0
result = ds.RNA.rawData.sum(axis=0).compute(
    nthreads=ds.nthreads,
)

Custom normalization functions must now accept and return ChunkedArray.

---

Storage and performance

New Zarr v3 writes provide:

• Memory-budgeted sharding
• Local and cloud storage profiles
• Feature-major countsT
• Remote-aware prefetch and local graph staging
• ANN indexes stored directly in Zarr
• Float32 normalized matrices
• Configurable memory and concurrency limits

ds = scarf.DataStore(
    "s3://bucket/data.zarr",
    storage_options={...},
    zarrProfile="cloud",
    mem_budget="8G",
    working_copies=8,
    nthreads=8,
)

countsT accelerates HVG statistics, marker search, and other feature-major reads.

Existing stores can optionally be repacked:

uv run python -m scarf.tools.repack_zarr \
    old.zarr \
    new.zarr \
    --profile fast_local

See Zarr internals for storage profiles, budgets, sharding, and countsT.

---

New plotting API

scarf.plotting replaces all legacy plotting entry points.

import scarf.plotting as splt

splt.embedding(
    ds,
    layout_key="RNA_UMAP",
    color_by="RNA_leiden_cluster",
    theme="notebook",
    legend_loc="auto",
)

The API includes:

• embedding and embedding_raster
• unified_embedding
• dotplot and matrixplot
• composition and distribution
• QC, graph, elbow, and HVG diagnostics
• Marker, cluster-tree, and pseudotime heatmaps

Plot functions return PlotResult, support shared themes and scales, and rasterize large embeddings automatically.

---

Harmony and reference mapping

Harmony now lives alongside PCA and LSI under scarf.embeddings.

ds.make_graph(
    feat_key="hvgs",
    harmonize=True,
    batch_columns=["batch"],
    harmony_params={"nclust": 30},
)

Batch-aware mapping references can be persisted and reused:

reference = ds.build_mapping_reference(
    from_assay="RNA",
    feat_key="hvgs",
    batch_columns=["batch", "donor"],
)

result = reference.map_query(
    query_assay,
    "query",
    "hvgs",
    query_batches=query_metadata[["batch"]],
)

Mapping references include provenance, batch metadata, corrected coordinates, and ANN contracts. Label-transfer evidence now supports weighted votes, entropy, margins, and conformal prediction sets.

---

Analysis improvements

Graphs and embeddings

• Corrected WNN affinity and unequal-neighbor handling
• Improved graph cache validation
• In-Zarr ANN persistence
• Deterministic incremental PCA without mutating input arrays
• Harmony parameters included in graph cache identity

Existing WNN graphs should be rebuilt.

Features and trajectories

• Faster HVG and marker calculations through countsT
• Genomic feature construction consolidated under scarf.features
• Connected-component policies for pseudotime
• Stricter source, sink, and regressor validation
• Structured pseudotime result objects
• Feature-profile aggregation and module clustering

Quality control and metrics

• Doublet detection through DataStore.run_doublet_detection
• Improved automatic filtering and cell-cycle scoring
• Dedicated label concordance, batch mixing, LISI, and silhouette metrics
• Safer LISI perplexity handling for small neighborhoods

---

Internal architecture

The implementation is now organized into concrete packages:

storage, matrix, utils
metadata, assay
neighbors, embeddings, clustering, trajectory
features, quality_control, metrics, mapping
readers, writers, merge
datastore
plotting

DataStore remains the main workflow API. Its methods now delegate to focused operation mixins, while reusable algorithms live in their owning domain packages.

Lazy public facades reduce initial import cost and avoid loading optional dependencies until needed.

See the architecture guide for package responsibilities and dependency rules.

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[parashardhapola]

Fixes #139
Closes #102
Closes #89