alterlab-zarr
Chunked, compressed N-dimensional arrays for cloud storage with Zarr — parallel I/O, S3/GCS integration, and NumPy/Dask/Xarray compatibility. Use when storing or reading large N-D scientific arrays, streaming chunked data to/from cloud object stores, or building large-scale scientific computing pipelines. Part of the AlterLab Academic Skills suite.
How do I install this agent skill?
npx skills add https://github.com/alterlab-ieu/alterlab-academic-skills --skill alterlab-zarrIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides comprehensive documentation and examples for using the Zarr library for chunked, compressed N-dimensional array storage. It is safe for its intended scientific computing use case. A low-severity finding for indirect prompt injection is noted as the skill processes external data without specific sanitization, which is typical for data analysis tools. It also involves standard Python package installations.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
Zarr Python
Overview
Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray. The guidance targets zarr-python 3.x (current 3.4.0), which writes the Zarr v3 format by default.
When to Use This Skill
Use this skill when the user wants to:
- Create, open, or write chunked and compressed N-D arrays (
zarr.create_array, groups, attributes). - Choose chunk shapes, sharding, and codecs (Zstandard, Blosc, Gzip) for an access pattern.
- Put arrays on S3/GCS or other object stores and read slices remotely (consolidated metadata, sharding).
- Convert HDF5/NumPy/NetCDF data to Zarr, or migrate Zarr v2-era code (
compressor=,synchronizer=,create_dataset) to v3.
Does NOT Trigger
| Scenario | Use Instead |
|---|---|
| Parallel or distributed computation over arrays already stored in Zarr (scheduling, out-of-core math) | alterlab-dask |
| Exploring an unknown data file and writing a structure/quality report | alterlab-eda |
Single-cell AnnData objects stored as .h5ad or AnnData-on-zarr | alterlab-anndata |
| Tabular analytics on large CSV/Parquet files | alterlab-polars |
Quick Start
Installation
uv pip install zarr
Zarr v3 (zarr>=3); zarr 3.2+ requires Python 3.12+ and NumPy 2 (the 3.1.x line still supports
Python 3.11). For cloud storage support, install the matching fsspec backend:
uv pip install s3fs # For S3
uv pip install gcsfs # For Google Cloud Storage
Basic Array Creation
import zarr
import numpy as np
# Create a 2D array with chunking and compression
z = zarr.create_array(
store="data/my_array.zarr",
shape=(10000, 10000),
chunks=(1000, 1000),
dtype="f4"
)
# Write data using NumPy-style indexing
z[:, :] = np.random.random((10000, 10000))
# Read data
data = z[0:100, 0:100] # Returns NumPy array
Core Workflow
- Create or open an array/group, picking a store appropriate to the environment (local, in-memory, ZIP, S3/GCS).
- Choose chunking aligned to your access pattern (aim for 1-10 MB chunks; rows-first → chunks span columns, and vice versa). This is the single biggest performance lever.
- Pick compression via
compressors=based on workload — Zstandard (the default), Blosc+LZ4 (fast), Gzip (max ratio);compressors=Noneto disable. Pass Blosc options as strings (shuffle="shuffle",cname="zstd"); theBloscShuffle/BloscCnameenums are deprecated since 3.1. - Read/write with NumPy-style indexing; resize/append as data grows.
- Scale out with Dask (lazy, out-of-core, parallel) or label with Xarray for climate/geospatial data.
- For cloud and many-array stores, consolidate metadata and consider sharding to cut object/file count. Consolidated metadata is a zarr-python convention, not yet part of the v3 spec (zarr warns), so other readers may ignore it.
# Minimal end-to-end
import zarr, numpy as np
z = zarr.create_array(store="data/my_array.zarr", shape=(10000, 10000),
chunks=(1000, 1000), dtype="f4")
z[:, :] = np.random.random((10000, 10000))
sub = z[0:100, 0:100] # returns a NumPy array
Routing — where to look
| You need… | Go to |
|---|---|
| Array create/open, read/write, resize/append, attributes, groups & hierarchies, consolidated metadata | references/array_operations.md |
| Chunk-size guidelines, aligning chunks to access patterns, sharding, compression codecs & tips | references/chunking_compression.md |
| Local / in-memory / ZIP / S3 / GCS stores and cloud best practices | references/storage_backends.md |
| NumPy / Dask / Xarray integration, thread- and process-safe parallel writes | references/integration.md |
| Performance checklist, profiling, common patterns (time series, large matrices, cloud-native, format conversion), troubleshooting | references/patterns_performance.md |
| Full API surface | references/api_reference.md |
Additional Resources
- Official Documentation: https://zarr.readthedocs.io/
- Zarr Specifications: https://zarr-specs.readthedocs.io/
- GitHub Repository: https://github.com/zarr-developers/zarr-python
- Related: Xarray (labeled arrays), Dask (parallel computing), NumCodecs (compression codecs)
Part of the AlterLab Academic Skills suite.
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/alterlab-ieu/alterlab-academic-skills/alterlab-zarr">View alterlab-zarr on skillZs</a>