Sources (rojak.core.sources)#

Describe where gridded input data lives, and open it as a single xarray.Dataset

This module only answers “which files/stores make up this dataset, and how are they opened”. Adapting the opened dataset into rojak’s variables (e.g. CATData) is the job of a MetData subclass, selected via rojak.datalib.registry.

A location is one of the following, distinguished by its type key in the configuration (a pydantic discriminated union, see DataLocation):

  • FilesLocation (type: files): every file matching a glob pattern under a root. This is the behaviour rojak has always had (data_dir + glob_pattern).

  • TemplateLocation (type: template): files named by a str.format() template over a time window, e.g. "{time:%Y}/{time:%m}/{time:%Y-%m-%d}.nc". Selects data by time rather than by folder layout, in the spirit of satpy/trollsift, ESMValTool’s input_dir templates and Pangeo Forge’s FilePattern.

  • ZarrLocation (type: zarr): a single (local or remote) zarr store.

Every root/store is an fsspec URL path, so a plain local path (~/data/era5), an explicit file:// URL and remote URLs (s3://, gs://, https://, memory://, …) are all handled the same way. Remote protocols need their fsspec implementation installed (e.g. s3fs, gcsfs) and can be given credentials or other options via storage_options.

Module Attributes

DataLocation

Where a gridded dataset lives, see the module docstring for the options

Functions

is_remote(urlpath)

Whether urlpath refers to a non-local filesystem

open_dataset(location, chunks[, engine, ...])

Lazily open location as a single dataset

resolve_paths(location)

Resolve a file-based location into the (sorted, for FilesLocation) list of files to open

Classes

FilesLocation(*[, storage_options, type, ...])

Every file under root matching glob_pattern

NetcdfEngine(*values)

xarray backend engine used to read netCDF files

TemplateLocation(*[, storage_options, type, ...])

Files under root named by template formatted with each time step in time_window

ZarrLocation(*[, storage_options, type, ...])

A single zarr store