prefect_managedfiletransfer.download_file_task
Classes
Functions
download_file_task
async
Task to download a single file from a remote source (SFTP, RClone, or LocalFileSystem) to a local path Args: source_block (TransferBlockType): The block representing the source connection. source_type (RemoteConnectionType): The type of the source connection. remote_asset (RemoteAsset): The remote asset to download. target_file_path (Path): The path where the file should be downloaded. update_only_if_newer_mode (bool): If true, skip files that are newer on the destination. overwrite (bool): If true, overwrite the file if it exists. mode (str): The transfer mode to use (e.g., Copy, Move). rclone_config (RCloneConfigSavedInPrefect | None): The RClone configuration to use for the download, if applicable. check_for_space (bool): If true, check if there is enough space on the destination. check_for_space_overhead (int): The overhead space to consider when checking for space in bytes. reference_date (datetime | None): The reference date to use for checking file modification times. Defaults to None, which uses the current time. Returns: AssetDownloadResult: The result of the download operation, including success status and any error messages.
Source code in prefect_managedfiletransfer/download_file_task.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 |
|
prefect_managedfiletransfer.list_remote_files_task
Classes
Functions
list_remote_files_task
async
Task to list remote files based on a matcher. Remote can be SFTP, RClone, or LocalFileSystem.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
source_block
|
TransferBlockType
|
The block representing the source connection. |
required |
source_type
|
RemoteConnectionType
|
The type of the source connection. |
required |
matcher
|
FileMatcher
|
The matcher defining the pattern to match files. |
required |
rclone_config
|
RCloneConfigSavedInPrefect | None
|
The RClone configuration to use for the remote connection, if applicable. |
None
|
reference_date
|
datetime | None
|
The reference date to use for filtering files. Defaults to None, which uses the current time. |
None
|
Returns:
Type | Description |
---|---|
list[RemoteAsset]
|
list[RemoteAsset]: A list of RemoteAsset objects that match the given pattern. |
Source code in prefect_managedfiletransfer/list_remote_files_task.py
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 |
|
prefect_managedfiletransfer.upload_file_task
Classes
Functions
upload_file_task
async
Task to upload a single file to a remote destination (local/SFTP/RClone remote).
Parameters:
Name | Type | Description | Default |
---|---|---|---|
source_remote_asset
|
RemoteAsset
|
The remote asset to upload. |
required |
destination_block
|
ServerWithBasicAuthBlock | ServerWithPublicKeyAuthBlock | LocalFileSystem | RCloneConfigFileBlock
|
The block representing the destination. |
required |
destination_type
|
RemoteConnectionType
|
The type of the destination connection. |
required |
target_file_path
|
Path
|
The path where the file should be uploaded. |
required |
update_only_if_newer_mode
|
bool
|
If true, skip files that are newer on the destination. |
required |
overwrite
|
bool
|
If true, overwrite the file if it exists. |
required |
mode
|
TransferType
|
The transfer mode to use (e.g., Copy, Move). |
required |
rclone_config
|
RCloneConfigFileBlock
|
The RClone configuration block to use for the upload. |
required |
check_for_space
|
bool
|
If true, check if there is enough space on the destination. |
required |
check_for_space_overhead
|
int
|
The overhead space to consider when checking for space in bytes. |
required |
reference_date
|
datetime
|
now() in UTC. The reference date to use for checking file modification times. Used in testing. |
required |
Source code in prefect_managedfiletransfer/upload_file_task.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 |
|