Move to Azure Blob without read permission fails on first try #6162
Comments
|
I think what is happening here is that the post copy check is failing Here rclone is attempting to read the MD5 sum etc off the object to check it got transferred OK. There isn't currently a workaround for this as rclone will attempt to re-read the metadata regardless of the setting of --ignore-checksum. Would it be possible to add HEAD objects permission to your SAS URL? That will let rclone check the objects got uploaded correctly. |
|
There is no HEAD permission in Azure Blob. Though there is a List permission and when listing blobs you can get the MD5 hash (according to this: https://stackoverflow.com/questions/67969357/azure-blob-content-md5-property-is-null-while-reading-it-using-azure-blob-python). The above result is actually with Write + List permission. (Without List permission the operation fails at an even earlier stage and does not complete at all). The follow up question is then, if rClone cannot read the MD5 sum etc to check it got transferred OK, how come it succeeds at attempt 2/3? " Is it so that it at the start of the second attempt, rClone does a LIST action to retrieve all metadata (md5, size, modification time etc) and therefore because of the List permission it works? If not, how can it be sure enough that it is transferred OK that it removes the local file? |
That is correct, yes. As you noted above, the md5sums are in the listing, so when rclone retries, it discovers the file is there with the correct checksum so it doesn't need to copy it. |
… no_head_object/add no_read_for_metadata) There are three reasons an option why one would avoid HEAD requests on azureblob, with this now implementing the latter one: - no_head_object: Don't do a HEAD *before* writing to (or reading from) an object, to reduce the number of transactions and thus increase performance. **This is quite complex and thus not implemented here, see s3.Object.Update()!** - no_head: Don't do a HEAD *after* writing to an object, for the same reason but at the cost of "increasing the chance for undetected upload failures". This already exists in azureblob, a small issue was fixed here. - no_read_for_metadata: Instead of doing a HEAD to get metadata, use a list operation, to avoid requiring READ permissions for immutable/append-only applications (this seems to be quite azureblob-specific) - one could also use both no_head_object and no_head together to achieve this, but then uploads wouldn't be verified. This hence fixes rclone#6162 and rclone#7027, as well as an as of now unreported issue where the --no-head option wouldn't work at all with a prefix set on the remote.
… no_head_object/add no_read_for_metadata) There are three reasons an option why one would avoid HEAD requests on azureblob, with this now implementing the latter one: - no_head_object: Don't do a HEAD *before* writing to (or reading from) an object, to reduce the number of transactions and thus increase performance. **This is quite complex and thus not implemented here, see s3.Object.Update()!** - no_head: Don't do a HEAD *after* writing to an object, for the same reason but at the cost of "increasing the chance for undetected upload failures". This already exists in azureblob, a small issue was fixed here. - no_read_for_metadata: Instead of doing a HEAD to get metadata, use a list operation, to avoid requiring READ permissions for immutable/append-only applications (this seems to be quite azureblob-specific) - one could also use both no_head_object and no_head together to achieve this, but then uploads wouldn't be verified. Also fixes an as of now unreported issue where the --no-head option wouldn't work at all with a prefix set on the remote. Fixes rclone#6162 and rclone#7027
… no_head_object/add no_read_for_metadata) There are three reasons an option why one would avoid HEAD requests on azureblob, with this now mainly implementing the latter one: - no_head_object: Don't do a HEAD *before* writing to (or reading from) an object, to reduce the number of transactions and thus increase performance. **This already exists in azureblob, a small issue was fixed here where using a prefix would still require a HEAD request to the prefix itself.** - no_head: Don't do a HEAD *after* writing to an object, for the same reason but at the cost of "increasing the chance for undetected upload failures". **This is still not implemented for azureblob yet and quite complex, see s3.Object.Update()!** - no_read_for_metadata: Instead of doing a HEAD to get metadata, use a list operation, to avoid requiring READ permissions for immutable/append-only applications (this seems to be quite azureblob-specific) - one could also use both no_head_object and no_head together to achieve this, but then uploads wouldn't be verified. Fixes rclone#6162 and rclone#7027
What is the problem you are having with rclone?
I have set up an Azure Blob storage with a SAS key with Write but WITHOUT Read permission. I then try to move local files to that storage. rClone gives an error message that it failed, but then it says afterwards that it worked and deletes the local file. I've checked and the data is correctly uploaded so the issue more along the lines of that the error message creates confusion.
If changing move to copy, there is no error message.
What is your rclone version (output from
rclone version)rclone v1.58.0
Which cloud storage system are you using? (e.g. Google Drive)
Azure Blob Storage
The command you were trying to run (e.g.
rclone copy /tmp remote:tmp)Note, folder C:\Users\Tomas\dev\test contains one text file "test C.txt" containing a small dummy string.
A log from the command with the
-vvflag (e.g. output fromrclone -vv copy /tmp remote:tmp)How to use GitHub
The text was updated successfully, but these errors were encountered: