Running a recursive mc undo with time limitations?

[jaw-sh]

Hello,

I ran a sync command instead of copy and terminated it when I realized it was also deleting files. My buckets are versioned so the files are still there, but they are the last version.

I would like to undo any DELETE actions which happened between two points of time. Is there any way to accomplish this, or will I have to write a script?

Thanks!

[Bjohnson131]

The Go api for Minio supports custom SELECT expressions for restore operations (https://github.com/minio/minio-go/blob/v7.0.69/api-restore.go#L85). One may be able to write an expression to select files that were deleted during this custom time frame.

Let me tinker a bit, and see what is currently supported. I'll get back to you.

[Bjohnson131]

Hey @jaw-sh. I was able to un-delete versioned objects in my Minio installation with a single bash command. In short, the best tool to use for this job is the AWS cli- minio's CLI doesn't provide the functionality that is needed.

Using the object versioning commands provided by the list-object-versions command, we can delete the 'delete marker' on all of the objects.
I used these docs as a reference: https://repost.aws/knowledge-center/s3-undelete-configuration, specifically from the Remove the delete marker on thousands of objects section.

here's the monster of a command that you can use here:

aws s3api list-object-versions --bucket bk2 --endpoint-url http://localhost:9110 --query 'DeleteMarkers[?IsLatest==`true` && LastModified > `2024-04-24T22:38:02.501000+00:00` && LastModified < `2024-04-24T22:38:03.478000+00:00`].[Key, VersionId]' | jq -r '.[] | "--key '\''" + .[0] + "'\'' --version-id " + .[1]' | xargs -L1 aws s3api delete-object --bucket bk2 --endpoint-url http://localhost:9110

I'll explain.

list the version metadata

aws s3api list-object-versions --bucket bk2 --endpoint-url http://localhost:9110
this will list versioning data for all objects in the bk2 bucket.

query for deletion markers between a certain date

--query 'DeleteMarkers[?IsLatest==`true` && LastModified > `2024-04-24T22:38:02.501000+00:00` && LastModified < `2024-04-24T22:38:03.478000+00:00`].[Key, VersionId]'
We're looking for events that are DeleteMarkers, and that have are the latest version of the objects (IsLatest), then we look look for a LastModified property between 22:38:02.501 and 22:38:03.478 on 2024-04-24, this is the time that the deletion was comitted to the server. finally, .[Key, VersionId] selects these two properties only from the json.

remove the deletion marker from minio

| jq -r '.[] | "--key '\''" + .[0] + "'\'' --version-id " + .[1]' | xargs -L1 aws s3api delete-object --bucket bk2 --endpoint-url http://localhost:9110
This was copied directly from the first page linked above. Change the bk2 bucket, and the --endpoint-url as needed.

Some warnings

The --query is using a string comparison for LastModified NOT a date comparison. This will not work if you don't use the same timezone for all minio instances. my instances all used UTC by default.

Make sure to run the first list-object-versions command to validate that your selection set is correct before running the rest of the command. you can remove the .[Key, VersionId] to get the full data set.

the xargs command will take a while. you may want to test with parallel workers.

I highly recommend you try a dry run, as the version of minio I have deployed may have features that your deployment does not.

[jaw-sh]

The command aws s3api delete-object returns a screen that requires human input every time a deletion happens. I need to undelete a lot of files. I've read through their documentation but it doesn't look like there's any way to bypass this.

Similarly, aws s3api list-object-versions does not pipe results, it creates a complete list. There's a lot of directories at play and this would probably take hours.

I'm contemplating just running Command A into a text file overnight and then piping that into Command B which would run overnight as well.

[Bjohnson131]

hmm. I didn't see the prompt for deleting objects, it deletes without asking any input on my end.. I'm using 2.15.41:
Good news, I've buffed the script. using the aws s3api delete-objects, you can restore 1000 objects at a time.

I've re-formatted the ugly command up there into an uglier bash script:

#!/bin/bash

master_list="list.json"
count=1

filecount=1

bucket="bk2"
endpoint="http://localhost:9110"
findAfter="2023-01-01T22:38:02.501000+00:00"
findBefore="2025-01-01T22:38:03.478000+00:00"

aws s3api list-object-versions --bucket $bucket --endpoint-url $endpoint --query "DeleteMarkers[?IsLatest==\`true\` && LastModified > \`$findAfter\` && LastModified < \`findBefore\`].{Key: Key, VersionId: VersionId}" | jq -c '.[]' > $master_list

rm split*.json
printf "{\n\"Objects\": [" >> "split$filecount.json"
while IFS= read -r line
do

  if (( count % 1000 == 0 )); then  
    truncate -s-2 split$filecount.json
    printf "]\n}" >> "split$filecount.json"
    ((filecount++))
    printf "{\"Objects\": [" >> "split$filecount.json"
  fi

  echo "$line," >> "split$filecount.json"
  ((count++))
done < "$master_list"

truncate -s-2 split$filecount.json
printf "]\n}" >> "split$filecount.json"

running this will output files to your current directory labelled split1.json, split2.json, etc. Each has up to 1000 keys you can restore
You can accept these into the delete objects command:
aws s3api delete-objects --bucket bk2 --endpoint-url http://localhost:9110 --delete file://split1.json
you can run this for every splitn.json.

[jaw-sh]

In my tests this was extremely slow to the point of being unacceptable. There are over 3 million objects in an invalid state and after 10 minutes or so it had generated about 36 JSON files of the ~3600 needed. The format was correct, though.

I'm now wondering if I can use the MCLI to do this instead.

mcli undo minio/bucket/path --last 1 --action "delete"

I have all objects in an erroneous state in an array like [ [ "key", "version" ] ], and with MCLI I don't even need a version, so it's just looping this. I feel like this approach is also better because it's not deleting a version, it's creating a new one. I just hope that Minio has deduplication so this doesn't double up the physical file storage?

Unless you have any comments on that I'll try to proceed with that direction.

[Bjohnson131]

Ah, I was under the impression we're talking 10s of thousands of invalid objects, millions does indeed require a different solution, apologies.
if you have the keyset in the master list, you could do this, although be aware the mc undo doesn't create a new version, it removes the most recent version.

you can see this by creating an object, then deleting it in a versioned bucket.
aws s3api list-object-versions --bucket bk2 --endpoint-url http://localhost:9110 --prefix "path/to/your/deleted/file.png"
using this command you can see the versions.

then you can run mcli undo ..., and then the list-object-versions command again, and see that the delete marker was simply removed. It's the same effect as removing the delete marker through S3 API. If you're more comfortable with this method, I'd recommend it over the s3api way. IMO it's better to know your tools than to be able to use them.