tasks
importing.tasks
¤
Classes¤
DataImport
¤
Bases: PermissionsBase
Model to store the data imports.
This model stores the data imports, which are files with data that are uploaded to the system. The data is then processed asynchronously and stored in the database.
Attributes:
Name | Type | Description |
---|---|---|
station |
ForeignKey
|
Station to which the data belongs. |
format |
ForeignKey
|
Format of the data. |
rawfile |
FileField
|
File with the data to be imported. |
date |
DateTimeField
|
Date of submission of the data. |
start_date |
DateTimeField
|
Start date of the data. |
end_date |
DateTimeField
|
End date of the data. |
records |
IntegerField
|
Number of records in the data. |
observations |
TextField
|
Notes or observations about the data. |
status |
TextField
|
Status of the import. |
log |
TextField
|
Log of the data ingestion, indicating any errors. |
reprocess |
BooleanField
|
If checked, the data will be reprocessed. |
Functions¤
clean()
¤
Validate information and uploads the measurement data.
Source code in importing\models.py
104 105 106 107 108 109 110 111 112 113 114 115 116 |
|
Functions¤
clear_cache()
¤
Clear the cache.
Source code in importing\tasks.py
44 45 46 47 48 49 |
|
ingest_data(data_import_pk)
¤
Initiate the ingestion of data into the DB.
If the status of the data import is "not queued", the request is processed. The data loaded and saved to the database. The status is updated to completed or failed depending on the outcome.
Source code in importing\tasks.py
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
|
save_temp_data_to_permanent(data_import)
¤
Function to pass the temporary import to the final table.
Uses the data_import_temp object only to get all required information from its fields.
This function carries out the following steps:
- Bulk delete of existing data between two times on a given measurement table for the station in question.
- Bulk create to add the new data from the uploaded file.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
data_import_temp |
DataImportTemp object. |
required |
Source code in importing\functions.py
262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 |
|
update_variables_for_station(*station_codes)
¤
Update the variables for the given station codes.
The variables are updated based on the measurements associated with the station. The variables are saved as a comma-separated string in the variables field of the station model.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
station_codes |
tuple[str]
|
Station codes for which to update the variables. If not provided, all station codes with measurements are considered. |
()
|
Source code in station\functions.py
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
|