prefect_managedfiletransfer.transfer_files_flow
Classes
Functions
transfer_files_flow
async
Transfers files from a source to a destination based on the provided matchers and mapping. Args: source_block: The source block to transfer files from. destination_block: The destination block to transfer files to. source_file_matchers: List of file matcheing patterns to find and filter files in the source. path_mapping: List of file-to-folder mappings for transferring files. destination_folder: The path of the folder in destination_block where files will be transferred. update_only_if_newer_mode: If true, skip files that are newer on the destination. overwrite: If true, overwrite existing files in the destination. check_for_space: If true, check if there is enough space on the destination before transferring. check_for_space_overhead: Amount of extra space to reserve on the destination (in bytes). mode: Copy or Move transfer mode. reference_date: defaults to now() in UTC - used to filter files based on modification time, and for pattern replacement in file names Returns: A list of the Paths of transferred files.
Source code in prefect_managedfiletransfer/transfer_files_flow.py
58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 |
|
prefect_managedfiletransfer.upload_file_flow
Classes
Functions
upload_file_flow
async
Publish a single file to a destination, e.g. upload an image to a website or copy a file to a local shared public folder
Parameters:
Name | Type | Description | Default |
---|---|---|---|
source_folder
|
Path
|
The folder where the file to upload is located. |
required |
pattern_to_upload
|
str
|
The pattern of the file to upload, e.g. "*.jpg". |
required |
destination_file
|
Path
|
The destination file path where the file will be uploaded. |
required |
destination_block_or_blockname
|
ServerWithBasicAuthBlock | ServerWithPublicKeyAuthBlock | LocalFileSystem | RCloneConfigFileBlock | str
|
The destination block or block name where the file will be uploaded. |
required |
update_only_if_newer_mode
|
bool
|
If true, skip files that are newer on the destination. |
False
|
mode
|
TransferType
|
The transfer mode to use, e.g. Copy or Move. |
Copy
|
overwrite
|
bool
|
If true, overwrite the file if it already exists at the destination. |
False
|
Source code in prefect_managedfiletransfer/upload_file_flow.py
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 |
|