API docs

Core data model

Key([offsets, vars, indices])

Key for keeping track of chunks of a distributed Dataset.

Reading and writing data

open_zarr(store, **kwargs)	Returns a lazily indexable xarray.Dataset and chunks from a Zarr store.
DatasetToChunks(dataset[, chunks, ...])	Split one or more xarray.Datasets into keyed chunks.
ChunksToZarr(store[, template, zarr_chunks, ...])	Write keyed chunks to a Zarr store in parallel.
DatasetToZarr(dataset, store[, zarr_chunks])	Write an entire xarray.Dataset to a Zarr store.
make_template(dataset[, lazy_vars])	Make a lazy Dask xarray.Dataset for use only as a template.
replace_template_dims(template, ...)	Replaces dimension(s) in a template with updates coordinates and/or sizes.
setup_zarr(template, store[, zarr_chunks, ...])	Setup a Zarr store.
validate_zarr_chunk(key, chunk, template[, ...])	Check a chunk for consistency against the given template.
write_chunk_to_zarr(key, chunk, store, template)	Write a single Dataset chunk to Zarr.

Aggregation

Mean(dim[, skipna, dtype, fanout, pre_aggregate])	Calculate the mean over one or more distributed dataset dimensions.
Mean.Globally([dim, skipna, dtype, fanout, ...])	Calculate global mean over a pcollection of xarray.Dataset objects.
Mean.PerKey([dim, skipna, dtype, fanout, ...])	Calculate per-key mean over a pcollection of (hashable, Dataset).
MeanCombineFn([sum_and_count, finalize])	CombineFn for computing an arithmetic mean of xarray.Dataset objects.

Rechunking

ConsolidateChunks(target_chunks)	Consolidate existing chunks across offsets into bigger chunks.
ConsolidateVariables([label])	Consolidate existing chunks across variables into bigger chunks.
SplitChunks(target_chunks)	Split existing chunks into smaller chunks.
SplitVariables()	Split existing chunks into a separate chunk per data variable.
Rechunk(dim_sizes, source_chunks, ...[, ...])	Rechunk to an arbitrary new chunking scheme with bounded memory usage.

Utility transforms

ValidateEachChunk([label])

Check that keys and dataset(s) are valid for xarray-beam transforms.

Utility functions

offsets_to_slices(offsets, sizes[, base])	Convert offsets into slices with an optional base offset.
validate_chunk(key, datasets)	Verify that a key and dataset(s) are valid for xarray-beam transforms.
consolidate_chunks(inputs[, combine_kwargs])	Consolidate chunks across offsets into (Key, Dataset) pairs.
consolidate_variables(inputs[, merge_kwargs])	Consolidate chunks across distinct variables into (Key, Dataset) pairs.
consolidate_fully(inputs, *[, merge_kwargs, ...])	Consolidate chunks via merge/concat into a single (Key, Dataset) pair.
split_chunks(key, dataset, target_chunks)	Split a single (Key, xarray.Dataset) pair into many chunks.
split_variables(key, dataset)	Split a single (Key, xarray.Dataset) pair into separate variables.
in_memory_rechunk(inputs, target_chunks)	Rechunk in-memory pairs of (Key, xarray.Dataset).

High-level interface

Warning

The high-level interface is experimental and offers no backwards compatibility guarantees.

Dataset(template, chunks, split_vars, ptransform)	Experimental high-level representation of an Xarray-Beam dataset.
Dataset.from_zarr(path, *[, chunks, ...])	Create an xarray_beam.Dataset from a Zarr store.
Dataset.to_zarr(path, *[, ...])	Write this dataset to a Zarr file.
Dataset.from_xarray(source, chunks, *[, ...])	Create an xarray_beam.Dataset from an xarray.Dataset.
Dataset.from_ptransform(ptransform, *, ...)	Create an xarray_beam.Dataset from a Beam PTransform.
Dataset.collect_with_direct_runner()	Collect a dataset in memory by writing it to a temp file.
Dataset.map_blocks(func, *[, template, ...])	Map a function over the chunks of this dataset.
Dataset.rechunk(chunks[, split_vars, ...])	Rechunk this Dataset.
Dataset.split_variables(*[, label])	Split variables in this Dataset into separate chunks.
Dataset.consolidate_variables(*[, label])	Consolidate variables in this Dataset into a single chunk.
Dataset.mean([dim, skipna, dtype, label])	Compute the mean of this Dataset using Beam combiners.
Dataset.head(*[, label])	Return a Dataset with the first N elements of each dimension.
Dataset.pipe(func, *args, **kwargs)	Apply a function to this dataset with method-chaining syntax.
normalize_chunks(chunks, template[, ...])	Normalize chunks for a xarray.Dataset.