tables

importing.tables

Classes

DataImport

Bases: PermissionsBase

Model to store the data imports.

This model stores the data imports, which are, often, files with data that are uploaded to the system. The data is then processed asynchronously and stored in the database.

Attributes:

Name	Type	Description
station	ForeignKey	Station to which the data belongs.
format	ForeignKey	Format of the data.
rawfile	FileField	File with the data to be imported.
date	DateTimeField	Date of submission of the data.
start_date	DateTimeField	Start date of the data.
end_date	DateTimeField	End date of the data.
records	IntegerField	Number of records in the data.
observations	TextField	Notes or observations about the data.
status	TextField	Status of the import.
log	TextField	Log of the data ingestion, indicating any errors.

Functions

clean()

Validate information and uploads the measurement data.

Source code in importing/models.py

def clean(self) -> None:
    """Validate information and uploads the measurement data."""
    tz = self.station.timezone
    if not tz:
        raise ValidationError("Station must have a timezone set.")

    if self.origin.origin == "Thingsboard" and not self.format.thingsboard:
        raise ValidationError("Ensure a Thingsboard-specific format is specified.")

DataImportTable

Bases: Table

MapLayerImport

Bases: PermissionsBase

Functions

clean()

Validate the uploaded GeoTIFF and its transformed lon/lat bounds.

Source code in importing/models.py

def clean(self) -> None:
    """Validate the uploaded GeoTIFF and its transformed lon/lat bounds."""

    if not self.file:
        return

    import rasterio
    from rasterio.warp import transform_bounds

    file_obj = self.file.file
    try:
        with rasterio.MemoryFile(file_obj.read()) as memfile:
            with memfile.open() as dataset:
                if dataset.count == 0:
                    raise ValidationError(
                        {"file": "File contains no raster bands."}
                    )
                if dataset.crs is None:
                    raise ValidationError(
                        {"file": "File has no coordinate reference system (CRS)."}
                    )

                left, bottom, right, top = transform_bounds(
                    dataset.crs,
                    "EPSG:4326",
                    dataset.bounds.left,
                    dataset.bounds.bottom,
                    dataset.bounds.right,
                    dataset.bounds.top,
                )

                if not (-180 <= left <= 180 and -180 <= right <= 180):
                    raise ValidationError(
                        {
                            "file": (
                                "GeoTIFF longitude values are out of range "
                                "[-180, 180]."
                            )
                        }
                    )
                if not (-90 <= bottom <= 90 and -90 <= top <= 90):
                    raise ValidationError(
                        {
                            "file": (
                                "GeoTIFF latitude values are out of range "
                                "[-90, 90]."
                            )
                        }
                    )
    except ValidationError:
        raise
    except Exception as e:
        raise ValidationError(
            {"file": f"Could not open as a valid GeoTIFF or read coordinates: {e}"}
        )
    finally:
        file_obj.seek(0)

MapLayerImportTable

Bases: Table

ThingsboardImportMap

Bases: PermissionsBase

Model to store Thingsboard device mappings to station variables.

This model maps Thingsboard devices to specific variables at stations, allowing data from IoT devices to be imported and associated with the correct station and variable combinations.

Attributes:

Name	Type	Description
tb_variable	CharField	Name of the variable in Thingsboard.
variable	ForeignKey	The existing variable in Paricia associated with this mapping.
tb_device_name	CharField	The name of the device in Thingsboard.
station	ForeignKey	The name of the corresponding station in Paricia.

Functions

clean()

Validate that the variable is valid for the station.

Source code in importing/models.py

def clean(self) -> None:
    """Validate that the variable is valid for the station."""
    try:
        station = self.station
    except ThingsboardImportMap.station.RelatedObjectDoesNotExist:
        raise ValidationError({"station": "Station is required."})

    try:
        variable = self.variable
    except ThingsboardImportMap.variable.RelatedObjectDoesNotExist:
        raise ValidationError({"variable": "Variable is required."})

    # Check if the variable is valid for the station through SensorInstallation
    if not SensorInstallation.objects.filter(
        variable=variable, station=station
    ).exists():
        raise ValidationError(
            {
                "variable": f"Variable '{variable}' is not configured for"
                f" station '{station}' via a sensor installation."
            }
        )

ThingsboardImportMapTable

Bases: Table