XarrayZarrArray¶

Header file: <libs/zarr/xarray_zarr_array.hpp> [source]

template<typename Store> inline void write_zattrs_json(Store &store, std::string_view name, std::string_view attrs)¶

Write attributes string to a store under a .zattrs key.

Write some data under .zattrs key in store for an array called ‘name’. The key and attrs data could be anything, but for example .zattrs could be a json file in a file system store (see FSStore) for the extra metadata which must exist in order to make Xarray and netCDF happy when opening a Zarr dataset, e.g. by naming the dimensions of the “{"_ARRAY_DIMENSIONS": ["dimension_name"]}”;.

Template Parameters:

Store – The type of the store object where the metadata will be written.

Parameters:

store – The store object where the metadata will be written.
name – The name under which the .zarray key will be stored in the store.
metadata – The metadata to write for the .zarray key.

inline std::vector<size_t> reduced_arrayshape_from_dims(const std::unordered_map<std::string, size_t> &datasetdims, const std::vector<std::string> &dimnames)¶

Calculate the reduced array shape of an array given the name of its dimensions and the dataset’s dimensions.

Given the dimensions of a dataset and the names of the dimensions of an array, this function calculates the reduced array shape by extracting the sizes of the dimensions from the dataset which correspond to the provided dimension names for all except for the outermost dimension of the array.

Parameters:

datasetdims – An unordered map containing the dimensions of the dataset.
dimnames – A vector containing the names of the dimensions of the array (ordered from outermost->innermost).

Returns:

A vector of size_t representing the reduced array shape.

inline std::string vecstr_to_string(const std::vector<std::string> &dims)¶

Converts vector of strings, e.g. for names of dimensions,into a single list written as a string.

Parameters:: dims – The vector of strings to be converted.
Returns:: A string representing the converted list.

inline std::string make_xarray_metadata(const std::string_view units, const std::string_view dtype, const double scale_factor, const std::vector<std::string> &dimnames)¶

Make string of array attributes metadata for .zattrs json which is used to make zarr array compatible with Xarray and NetCDF.

scale_factor is only stored in metadata if type of array is floating point (dtype “f”) or complex floating point (dtype “c”).

Parameters:

units – The units of the array’s coordinates.
scale_factor – The scale factor of data.
dimnames – The names of each dimension of the array.

Returns:

A string representing the metadata.

inline std::string make_xarray_metadata(const std::string_view units, const std::string_view dtype, const double scale_factor, const std::vector<std::string> &dimnames, const std::string_view sampledimname)¶

Make string of array attributes metadata for .zattrs json which is used to make array for a raggedcount variable in Zarr compatible with Xarray and NetCDF.

scale_factor is only stored in metadata if type of array is floating point (dtype “f”) or complex floating point (dtype “c”).

Parameters:

units – The units of the array’s coordinates.
scale_factor – The scale factor of data.
dimnames – The names of each dimension of the array.
sampledimname – The name of the dimension the ragged count samples.

Returns:

A string representing the metadata.

template<typename Store, typename T> class XarrayZarrArray¶

Zarr array with additional metadata and functions to constrain the shape of array to the shape of its dimensions in a dataset in order to ensure Zarr array is compatibile with NetCDF and Xarray conventions.

Template Parameters:

Store – The type of the store object where the array will be stored.
T – The data type of the array.

Public Functions

inline XarrayZarrArray(Store &store, const std::unordered_map<std::string, size_t> &datasetdims, const std::string_view name, const std::string_view units, const std::string_view dtype, const double scale_factor, const std::vector<size_t> &chunkshape, const std::vector<std::string> &dimnames)¶

Constructs a new XarrayZarrArray object.

Parameters:

store – The store where the array will be stored.
datasetdims – Dictionary like object for the dimensions of the dataset.
name – The name of the array.
units – The units of the array data.
dtype – The data type of the array.
scale_factor – The scale factor of array data.
chunkshape – The shape of the array chunks.
dimnames – The names of each dimension of the array (in order outermost->innermost).

Constructs a new XarrayZarrArray object with additional variable called “sample_dimension” in the metadata .zattrs json and initially no set arrayshape.

Parameters:

store – The store where the array will be stored.
datasetdims – Dictionary like object for the dimensions of the dataset.
name – The name of the array.
units – The units of the array data.
dtype – The data type of the array.
scale_factor – The scale factor of array data.
chunkshape – The shape of the array chunks.
dimnames – The names of each dimension of the array (in order outermost->innermost).
sampledimname – The name of the dimension the ragged count samples.

inline ~XarrayZarrArray()¶

inline std::unordered_map<std::string, size_t> get_arraydims() const¶

Returns the name and size of the dimensions of the array (unordered).

Returns:: An unordered map containing the current dimensions of the array.

inline void write_to_array(const viewh_buffer h_data)¶

Writes data from Kokkos view in host memory to chunks of a Zarr array in a store via a buffer. Function does not write metadata to zarray .json file.

Calls ZarrArray’s write_to_array function to write data from Kokkos view in host memory to chunks of a Zarr array in a store.

Parameters:: h_data – The data in a Kokkos view in host memory which should be written to the array in a store.

inline void write_to_array(const T data)¶

Writes 1 data element to a Zarr array in a store. Function does not write metadata to zarray .json file.

Calls ZarrArray’s write_to_array function to write data to a Zarr array in a store (in chunks via a buffer).

Parameters:: data – The data element which should be written to the array in a store.

inline void write_arrayshape(const std::unordered_map<std::string, size_t> &datasetdims)¶

Sets shape of array along each dimension to be the same size as each of its dimensions according to the dataset.

The order of the dimensions in the array’s shape is the order of dimensions in dimnames (outermost -> innermost). Setting the shape to be conistent with the size of the dataset’s dimensions makes zarr array also consistent with Xarray and NetCDF conventions. If chunks have been written since last writing of the arrayshape, and the shape of the array has changed, then function also overwrites the .zarray json file with metadata containing the new shape of the array.

Parameters:: datasetdims – Dictionary like object for the dimensions of the dataset.

inline void write_ragged_arrayshape()¶

Sets shape of array along each dimension to be the same a expected for a 1-D ragged array.

Expected shape is 1-D array with size of the total number of elements written to a zarr array.

If chunks have been written since last writing of the arrayshape, then function also overwrites the .zarray json file with metadata containing the new shape of the array.

Private Types

using viewh_buffer = Buffer<T>::viewh_buffer¶

Private Functions

inline bool set_arrayshape(const std::unordered_map<std::string, size_t> &datasetdims)¶

Sets shape of array along each dimension to be the same size as each of its dimensions according to the dataset. Returns boolean for whether shape has changed (true) or not (false).

The order of the dimensions in the array’s shape is the order of dimensions in dimnames (outermost -> innermost). Setting the shape to be conistent with the size of the dataset’s dimensions makes zarr array also consistent with Xarray and NetCDF conventions. Boolean returns true if the shape of the array along any of its dimensions has changed.

Parameters:: datasetdims – Dictionary like object for the dimensions of the dataset.
Returns:: bool = true if arrayshape along any of its dimensions has changed, false otherwise.

inline bool set_ragged_arrayshape()¶

Sets shape of array along each dimension to be the same as shape according to zarr.

Useful when writing a ragged arrray in a dataset (meaning length of dimensions if not length of array)

Private Members

ZarrArray<Store, T> zarr¶: zarr array in store

std::vector<std::string> dimnames¶: ordered list of names of each dimenion of array

std::vector<size_t> arrayshape¶: current size of the array along each of its dimensions

size_t last_totnchunks¶: Number of chunks of array since arrayshape last written