mc stat info and minio disk data are not synchronized? #19715

Amber-976 · 2024-05-10T10:13:46Z

Amber-976
May 10, 2024

Question description

Question 1

When the object expires, mc stat can't get the object, but the minio drive corresponding disk directory of the object and xl.meta still exist.

[root@minio-1 dx]# ls  -l   /export5/collect-ngaf-file-bucket0/20240508_NGAF_SECEVENT_blob_s_DB3EF0D2_data_MTA1NQ==_1715158837/
total 4
-rw-r--r-- 1 root root 443 May  8 17:04 xl.meta
[root@minio-1 dx]# mc stat myminio/collect-ngaf-file-bucket0/20240508_NGAF_SECEVENT_blob_s_DB3EF0D2_data_MTA1NQ==_1715158837
mc: <ERROR> Unable to stat `myminio/collect-ngaf-file-bucket0/20240508_NGAF_SECEVENT_blob_s_DB3EF0D2_data_MTA1NQ==_1715158837`. Object does not exist.

At https://min.io/docs/minio/kubernetes/upstream/operations/concepts/scanner.html, ilm of expiration is not immediate. And my question is, when the expiration time is up, the expired object file is still on the instant disk, but when the get or stat request is executed, minio will tell the object does not exist. Is it normal?

Question 2

The bucket expiration time is set to one day, and the lifecycle rules are as follows:

{
 "status": "success",
 "target": "myminio/collect-ngaf-file-bucket0",
 "config": {
  "Rules": [
   {
    "Expiration": {
     "Days": 1
    },
    "ID": "cot1mnbte3cv7kish05g",
    "Status": "Enabled"
   },
   {
    "Expiration": {
     "Days": 1
    },
    "ID": "cot1obbte3cn7cmrpseg",
    "Status": "Enabled"
   }
  ]
 },
 "updatedAt": "0001-01-01T00:00:00Z"
}

The distribution of the data inside the bucket is as follows:

[root@minio-1 dx]# mc stat  myminio/collect-ngaf-file-bucket0
Name      : collect-ngaf-file-bucket0
Date      : 2024-05-09 20:36:43 CST 
Size      : N/A    
Type      : folder 

Properties:
  Versioning: Un-versioned
  Location: us-east-1
  Anonymous: Disabled
  ILM: Enabled

Usage:
      Total size: 9.9 GiB
   Objects count: 583,855
  Versions count: 0

Object sizes histogram:
   486964 object(s) less than 1024 bytes
    96891 object(s) between 1024 bytes and 1 MB
        0 object(s) between 1 MB and 10 MB
        0 object(s) between 10 MB and 64 MB
        0 object(s) between 64 MB and 128 MB
        0 object(s) between 128 MB and 512 MB
        0 object(s) greater than 512 MB

When the expiration time has passed 10 hours, there are 400,000+ small objects left, which is too many objects to use ls. The slow removal of stale objects may have something to do with the speed of the scanner. I change scanner speed=fast to see that the slope of the monitored freeinodes does not change.
Will minio be very slow if there are lots of small objects in the bucket? In other words, if there are VFS or other hardware limitations, no matter how you change scanner speed, the lifetime deletion rate will not improve, right?

Answered by klauspost

May 10, 2024

Q1. Yes this is normal.

Q2: Not correct. You can observe scanner speed with mc admin scanner info ALIAS.

View full answer

klauspost · 2024-05-10T10:18:03Z

klauspost
May 10, 2024
Collaborator

Q1. Yes this is normal.

Q2: Not correct. You can observe scanner speed with mc admin scanner info ALIAS.

1 reply

Amber-976 May 10, 2024
Author

Q2: Not correct. You can observe scanner speed with mc admin scanner info ALIAS.

Yes.

[root@minio-1 dx]# mc admin scanner info myminio
+-------------------------------------------------------------------------------------------------------+
| Last full scan time:   0d3h4m; Estimated 234.17/month                                                 |
| Current cycle:         1108; Started: 2024-05-10 18:21:25.346260653 +0800 CST                         |
| Active drives:          10                                                                            |
| -------------------------------------- Last Minute Statistics --------------------------------------- |
| Objects Scanned:       119747 objects; Avg: 1.345ms; Rate: 172435680/day                              |
| Versions Scanned:      119747 versions; Avg: 17.782µs; Rate: 172435680/day                            |
| Versions Heal Checked: 0 versions; Avg: 0ms                                                           |
| Read Metadata:         119746 objects; Avg: 1.266ms, Size: 413 bytes/obj                              |
| ILM checks:            119747 versions; Avg: 13.171µs                                                 |
| Check Replication:     72 versions; Avg: 3.134µs                                                      |
| Verify Deleted:        0 folders; Avg: 0ms                                                            |
| Yield:                 4m43s total; Avg: 2.363ms/obj                                                  |
| ------------------------------------- Currently Scanning Paths -------------------------------------- |
| minio-1:9000/export1/collect-sta-log-file-bucket19/20240508_STA_TELNET_OPTION_LOG_1000100102_pop3...  |
| minio-1:9000/export2/collect-sta-log-file-bucket15/20240508_STA_TELNET_OPTION_LOG_1000100102_kerb...  |
| minio-1:9000/export3/collect-sta-log-file-bucket17/20240508_STA_TELNET_OPTION_LOG_1000100102_kerb...  |
| minio-1:9000/export4/collect-sta-log-file-bucket14/20240508_STA_TELNET_OPTION_LOG_1000100102_mong...  |
| minio-1:9000/export5/collect-sta-log-file-bucket14                                                    |
| minio-1:9000/export6/collect-sta-log-file-bucket17/20240508_STA_TELNET_OPTION_LOG_1000100102_kerb...  |
| minio-1:9000/export7/collect-sta-log-file-bucket18/20240508_STA_TELNET_OPTION_LOG_1000100102_pgsq...  |
| minio-1:9000/export8/collect-sta-log-file-bucket19/20240508_STA_TELNET_OPTION_LOG_1000100102_ntlm...  |
| minio-3:9000/export4/collect-sta-log-file-bucket16/20240508_STA_TELNET_OPTION_LOG_1000100102_http...  |
| minio-3:9000/export7/collect-sta-log-file-bucket15/20240508_STA_TELNET_OPTION_LOG_1000100102_icmp...  |
+-------------------------------------------------------------------------------------------------------+

Amber-976 · 2024-05-10T10:20:15Z

Amber-976
May 10, 2024
Author

And, why does mc stat myminio/bucket see a lot of object statistics? At this point, all objects in the bucket will have expired.
Can anyone give an explanation for the inconsistency of sensory phenomena? Thank you very much!

3 replies

klauspost May 10, 2024
Collaborator

Not sure what you mean. If you need a deeper investigation look into our subscriptions

Your numbers look fine. Your objects should expire within 3 hours.

Amber-976 May 10, 2024
Author

Thanks for your reply!
Twelve hours have passed since the expiration date, but the expired objects have not yet been removed from disk. I'm not sure if this is normal, the minio cluster had no read or write requests during this time.
mc admin trace --call storage myminio | grep DeleteVersion:

minio-2:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:57.176] /export8 collect-sta-log-file-bucket11 20240508_STA_TELNET_OPTION_LOG_1000100102_mongo.avro_NDg4OA==_1715159611.ext.http.data 1.527015ms
minio-2:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:57.185] /export2 collect-sta-log-file-bucket7 20240508_STA_DNS_LOG_1000100102_dns.kvd_MjUwMQ==_1715150584 737.605µs
minio-2:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:57.185] /export3 collect-sta-log-file-bucket7 20240508_STA_DNS_LOG_1000100102_dns.kvd_MjUwMQ==_1715150584 1.021689ms
minio-2:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:57.185] /export4 collect-sta-log-file-bucket7 20240508_STA_DNS_LOG_1000100102_dns.kvd_MjUwMQ==_1715150584 984.885µs
minio-2:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:57.185] /export1 collect-sta-log-file-bucket7 20240508_STA_DNS_LOG_1000100102_dns.kvd_MjUwMQ==_1715150584 1.00111ms
minio-2:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:57.195] /export5 collect-sta-log-file-bucket9 20240508_STA_TELNET_OPTION_LOG_1000100102_smb.avro_MjUzNg==_1715155083.ext.http.data 1.083861ms
minio-2:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:57.195] /export8 collect-sta-log-file-bucket9 20240508_STA_TELNET_OPTION_LOG_1000100102_smb.avro_MjUzNg==_1715155083.ext.http.data 1.032947ms
minio-2:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:57.195] /export6 collect-sta-log-file-bucket9 20240508_STA_TELNET_OPTION_LOG_1000100102_smb.avro_MjUzNg==_1715155083.ext.http.data 1.162039ms
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.124] /export4 collect-sta-log-file-bucket6 20240508_STA_TELNET_OPTION_LOG_1000100102_mysaql.avro_OTcxMw==_1715145654 1.032504ms
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.124] /export1 collect-sta-log-file-bucket6 20240508_STA_TELNET_OPTION_LOG_1000100102_mysaql.avro_OTcxMw==_1715145654 964.12µs
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.124] /export3 collect-sta-log-file-bucket6 20240508_STA_TELNET_OPTION_LOG_1000100102_mysaql.avro_OTcxMw==_1715145654 1.042634ms
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.125] /export2 collect-sta-log-file-bucket6 20240508_STA_TELNET_OPTION_LOG_1000100102_mysaql.avro_OTcxMw==_1715145654 959.762µs
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.129] /export4 collect-sta-log-file-bucket7 20240508_STA_TELNET_OPTION_LOG_1000100102_sybase.avro_MjE0Ng==_1715150602 892.892µs
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.129] /export3 collect-sta-log-file-bucket7 20240508_STA_TELNET_OPTION_LOG_1000100102_sybase.avro_MjE0Ng==_1715150602 1.008976ms
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.129] /export1 collect-sta-log-file-bucket7 20240508_STA_TELNET_OPTION_LOG_1000100102_sybase.avro_MjE0Ng==_1715150602 805.86µs
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.129] /export2 collect-sta-log-file-bucket7 20240508_STA_TELNET_OPTION_LOG_1000100102_sybase.avro_MjE0Ng==_1715150602 930.611µs
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.134] /export8 collect-sta-log-file-bucket8 20240508_STA_TELNET_OPTION_LOG_1000100102_066bd0c3-7c37-446a-84dc-ea8fb6fb9f4a.avro_NzE2MA==_1715148199.ext.http.data 740.484µs
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.134] /export5 collect-sta-log-file-bucket8 20240508_STA_TELNET_OPTION_LOG_1000100102_066bd0c3-7c37-446a-84dc-ea8fb6fb9f4a.avro_NzE2MA==_1715148199.ext.http.data 787.891µs
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.134] /export6 collect-sta-log-file-bucket8 20240508_STA_TELNET_OPTION_LOG_1000100102_066bd0c3-7c37-446a-84dc-ea8fb6fb9f4a.avro_NzE2MA==_1715148199.ext.http.data 855.087µs
minio-1:9000  [STORAGE storage.DeleteVersion] [2024-05-10T21:00:58.134] /export7 collect-sta-log-file-bucket8 20240508_STA_TELNET_OPTION_LOG_1000100102_066bd0c3-7c37-446a-84dc-ea8fb6fb9f4a.avro_NzE2MA==_1715148199.ext.http.data 963.219µs

At this point, the objects in the minio bucket should all expire and be deleted, as the duration has far exceeded the last full scan. However, the delete operation is performed all the time.

klauspost May 10, 2024
Collaborator

@Amber-976 Do you know How Amazon S3 calculates expiry based on an object's age?

The value that you specify is the number of days since object creation when the action will occur.

Amazon S3 calculates the time by adding the number of days specified in the rule to the object creation time and rounding the resulting time to the next day at midnight UTC. For example, if an object was created on 1/15/2014 at 10:30 AM UTC and you specify 3 days in a transition rule, then the transition date of the object would be calculated as 1/19/2014 00:00 UTC.

Amber-976 · 2024-05-11T02:16:48Z

Amber-976
May 11, 2024
Author

@Amber-976 Do you know How Amazon S3 calculates expiry based on an object's age?

yep. I mean the expiration time is 5/10/2014 00:00 UTC(5/10/2014 10:00 CST).

The 'Last full scan time' in mc admin scanner info ALIAS is about 3h. I think at 5/10/2014 10:00 UTC, the expired objects should all be removed and the Usage. Objects count in mc stat ALIAS/mybucket should be 0(since all objects are already expired). So, now I'm not sure if this is normal 😄

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mc stat info and minio disk data are not synchronized? #19715

{{title}}

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

mc stat info and minio disk data are not synchronized? #19715

Amber-976 May 10, 2024

Question description

Question 1

Question 2

Replies: 3 comments · 4 replies

klauspost May 10, 2024 Collaborator

Amber-976 May 10, 2024 Author

Amber-976 May 10, 2024 Author

klauspost May 10, 2024 Collaborator

Amber-976 May 10, 2024 Author

klauspost May 10, 2024 Collaborator

Amber-976 May 11, 2024 Author

Amber-976
May 10, 2024

Replies: 3 comments 4 replies

klauspost
May 10, 2024
Collaborator

Amber-976 May 10, 2024
Author

Amber-976
May 10, 2024
Author

klauspost May 10, 2024
Collaborator

Amber-976 May 10, 2024
Author

klauspost May 10, 2024
Collaborator

Amber-976
May 11, 2024
Author