MadisAmdarPreprocessor#

class rojak.datalib.madis.amdar.MadisAmdarPreprocessor(filepaths, glob_pattern=None)[source]#

Preprocesses raw MADIS ACARS netCDF files into filtered parquet files of turbulence-relevant observations

Each raw (gzip-compressed) netCDF file is decompressed, restricted to data_vars_for_turbulence, filtered to drop observations with invalid quality-control (quality_control_vars) or error (error_vars) flags, and written out as a parquet file (see apply_preprocessor()).

Parameters:

filepaths (list[Path])
glob_pattern (str | None)

__init__(filepaths, glob_pattern=None)[source]#

Parameters:

filepaths (Iterable[Path] | Path) – Files to preprocess. If glob_pattern is given, this is instead the root directory (or directories) to search for files matching glob_pattern within. If glob_pattern is None and this is a single path (not an iterable), it must point to an existing file.
glob_pattern (str | None) – If provided, glob pattern used to find files within filepaths, and relative_to_root_path is populated with each file’s directory relative to its root in filepaths. If None, filepaths is used as-is.

Raises:

ValueError – If glob_pattern is None and filepaths is a single path that is not an existing file

Return type:

None

Methods

`__init__`(filepaths[, glob_pattern])
`apply_preprocessor`(output_directory)	Preprocess every file in `filepaths` into a filtered parquet file under `output_directory`
`decompress_gz`(filepath)	Decompress a gzip-compressed file into a temporary file
`drop_invalid_error_data`(dataset)	Drop observations with a failing error flag on any of `error_vars`
`drop_invalid_qc_data`(dataset)	Drop observations with an invalid quality-control flag on any of `quality_control_vars`

Attributes

`data_vars_for_turbulence`
`dimension_name`
`error_vars`
`quality_control_vars`
`relative_to_root_path`
`filepaths`