scanner: Scan buckets asynchronously

[vadmeste]

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the [Apache 2 license] (https://www.apache.org/licenses/LICENSE-2.0).
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

• Scan buckets asynchrounously; multiple erasure sets scan in parallel, multiple disks in the same erasure set scan different buckets in parallel
• No data format is changed
• Cycle concept changed to be per bucket focused, the cycle is incremented ecah time a bucket is successfully or unsuccesfully scanned
• Next bucket to scan is chosen based on the lowest possible cycle number

Motivation and Context

How to test this PR?

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Unit tests added/updated
• Internal documentation updated
• Create a documentation update request here

[harshavardhana]

do you know how far this improves to share? @vadmeste

[klauspost]

@vadmeste Could you describe the changes in more detail?

[vadmeste]

@vadmeste Could you describe the changes in more detail?

So basically here, I tried to keep the change as minimal as possible; and I did not need to do any data format change as well;

So we will use everything as usual; the total usage; the data cache per erasure set; the data cache per bucket per erasure set; This should not be a problem even if we scan buckets in parallel; because after all, we will still scan one bucket per erasure set at a time, this should not cause any confusion in calculation at all.

The differences from the old code:

• Each bucket in each erasure set has its own scan cycle
• A bucket scan manager decides what is the next bucket to scan for a given erasure set
• erasure-server-pools.NSScanner call (and lower calls) will not return anymore, and will continously scan for buckets

Let's chat while you review this PR if you want

[harshavardhana]

I have rebased and pushed the changes @vadmeste

[shtripat]

looks good to me

[shtripat]

%s/Mark a bucket as done in a specific erasure set/Mark a bucket as scan started in a specific erasure set/g

[harshavardhana]

@vadmeste please rebase this and let the team know what is waiting on us for merging this PR.

[harshavardhana]

PTAL @klauspost need your review here

Is there any breakage in mc admin scanner status @vadmeste ?

[harshavardhana]

@vadmeste read the code again. Why do we need a change in data structure and why madmin-go needs to break here ?

[vadmeste]

@vadmeste read the code again. Why do we need a change in data structure and why madmin-go needs to break here ?

Basically after implementing the asynchronous bucket scanning, a cluster scan cycle will lose its meaning. The cycle will be per bucket per erasure set instead:

           (pool, set, bucket-name) => cycle

By the way, madmin-go wil still work with older MinIO deployments, later mc will show the approriate UI. If the cluster cycle information is available, then we know this is an old version and

[poornas]

Suggested change

	// Copyright (c) 2015-2023 MinIO, Inc.
	// Copyright (c) 2015-2024 MinIO, Inc.