Skip to content

Zarr

source module isofit.luts.stores.zarr

Zarr implementation for LUTs

Classes

Functions

  • is_ray_task — Utility to check if this environment is a ray task

  • cleanup — Checks the ISOFIT status attribute on a LUT and removes the file if it is incomplete.

  • shard_to_coord — Converts a shard index to shard coordinates (the slice of the full store for this shard).

  • calc_shards — Determine an optimal Zarr sharding strategy for a LUT and group points by shard.

source is_ray_task()

Utility to check if this environment is a ray task

source cleanup(path)

Checks the ISOFIT status attribute on a LUT and removes the file if it is incomplete.

Parameters

  • path : str — Path to the store to check

source shard_to_coord(group, shards, shape)

Converts a shard index to shard coordinates (the slice of the full store for this shard).

source calc_shards(grid, wl, chunk, storage='8gb', target_shards=None, scale=1)

Determine an optimal Zarr sharding strategy for a LUT and group points by shard.

This function computes candidate shard shapes that: - evenly divide the LUT dimensions - do not split chunks - approximately match a target storage size per shard

It then selects the shard shape whose number of chunks per shard is closest to the target, and groups LUT points into shard-aligned partitions.

Parameters

  • grid : dict[str, Iterable] — LUT grid definition as {dimension_name: values}. These define the non-wavelength dimensions of the LUT.

  • wl : iterable — Wavelength dimension values.

  • chunk : iterable of int — Chunk shape used for Zarr storage. Must align with the full LUT shape (wl + grid dimensions).

  • storage : str or float, default="8gb" — Storage size limit per shard. Determines how many chunks should be grouped into a shard while respecting the limit.

  • target_shards : int, optional — Target number of shards to produce. The closest sharding strategy to this will be selected.

  • scale : int, default=1 — Multiplier applied to the estimated chunk size. Useful when multiple arrays are written per chunk to better approximate the actual memory when in buffered mode.

Returns

  • best : np.array[int] — Selected shard shape, including the wavelength dimension as the first axis.

  • groups : dict[tuple[int, ...], list[tuple]] — Mapping from shard index (multi-dimensional shard ID) to the list of grid points (coordinate tuples) that fall within that shard.

  • coords : dict[tuple[int, ...], tuple[slice, ...]] — Mapping from shard index to the corresponding global array slice (including wavelength as the leading dimension). These slices can be used directly to write shard-aligned data into a Zarr array.

Raises

  • ValueError

source class CreateZarr(path, *args, mode='w', shards=None, buffered=False, shard_size=None, target_shards=None, **kwargs)

Bases : Create

Prepare a Zarr v3 LUT store

Parameters

  • path : str — Path for the LUT.

  • mode : str, default="w" — File mode to open with

  • shards : np.array, optional, default=None — Sharding strategy to use

  • buffered : bool, optional, default=False — Create intermediate buffer arrays to store data before flushing

  • shard_size : str | float, optional, default=None — Target size for each shard. May be either a string like "4gb" or a float representing number of bytes. If this parameter is given, the shards array will be calculated

  • target_shards : int, optional, default=None — Target number of shards to create. Useful for ensuring parallelization fully utilizes the available machine

  • wl : np.ndarray — The wavelength array.

  • grid : dict — The LUT grid, formatted as {str: Iterable}.

  • attrs : dict, defaults={} — Dict of dataset attributes, ie. {"RT_mode": "transm"}

  • consts : dict, optional, default={} — Dictionary of constant values. Appends/replaces current Create.consts list.

  • onedim : dict, optional, default={} — Dictionary of one-dimensional data. Appends/replaces to the current Create.onedim list.

  • alldim : dict, optional, default={} — Dictionary of multi-dimensional data. Appends/replaces to the current Create.alldim list.

  • zeros : List[str], optional, default=[] — List of zero values. Appends to the current Create.zeros list.

Methods

  • initialize — Initializes the LUT Zarr by prepopulating it with filler values.

  • flush_buffer — Needs to be manually called

  • reset_buffer — Resets the buffered data to ensure

  • queuePoint — Overrides the inherited queuePoint to enable flushing immediately for this subclass only

  • getAttr — Gets an attribute from the Zarr

  • setAttr — Sets an attribute in the Zarr

  • finalize — Finalizes the Zarr store by consolidating the metadata at the end to make reading it more efficient via xarray

source method CreateZarr.initialize(ret=False) → None | xr.Dataset

Initializes the LUT Zarr by prepopulating it with filler values.

Parameters

  • ret : bool, default=False — If True, returns the dataset instead of saving

Returns

  • None | xr.Dataset — The initialized dataset

source method CreateZarr.flush_buffer(slices)

Needs to be manually called

source method CreateZarr.reset_buffer()

Resets the buffered data to ensure

source method CreateZarr.queuePoint(*args, **kwargs)

Overrides the inherited queuePoint to enable flushing immediately for this subclass only

source method CreateZarr.getAttr(key: str) → Any

Gets an attribute from the Zarr

Parameters

  • key : str — Key to get

Returns

  • any | None — Retrieved attribute from Zarr, if it exists

source method CreateZarr.setAttr(key: str, value: Any) → None

Sets an attribute in the Zarr

Parameters

  • key : str — Key to set

  • value : any — Value to set

source method CreateZarr.finalize(*args, **kwargs)

Finalizes the Zarr store by consolidating the metadata at the end to make reading it more efficient via xarray