Add sufficient deadlines and countermeasures to handle hung node scenario #19688
+245
−120
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
harshavardhana
force-pushed
the
pause-container-test
branch
2 times, most recently
from
351b2af
to
5f9fec9
Compare
harshavardhana
force-pushed
the
pause-container-test
branch
4 times, most recently
from
118c7b2
to
3d473d9
Compare
krisis
approved these changes
May 14, 2024
harshavardhana
force-pushed
the
pause-container-test
branch
3 times, most recently
from
01aab05
to
23766c5
Compare
Comment on lines
-69
to
+71
| peersLogOnceIf(context.Background(), err, nodeName) | ||
| if xnet.IsNetworkOrHostDown(err, false) { | ||
| network[nodeName] = string(madmin.ItemOffline) | ||
| } else if xnet.IsNetworkOrHostDown(err, true) { | ||
| network[nodeName] = "connection attempt timedout" | ||
| } |
There was a problem hiding this comment.
To differentiate between actually "offline v/s timedout"
Comment on lines
-106
to
-117
| cli.DurationFlag{ | ||
| Name: "conn-client-read-deadline", | ||
| Usage: "custom connection READ deadline for incoming requests", | ||
| Hidden: true, | ||
| EnvVar: "MINIO_CONN_CLIENT_READ_DEADLINE", | ||
| }, | ||
| cli.DurationFlag{ | ||
| Name: "conn-client-write-deadline", | ||
| Usage: "custom connection WRITE deadline for outgoing requests", | ||
| Hidden: true, | ||
| EnvVar: "MINIO_CONN_CLIENT_WRITE_DEADLINE", | ||
| }, |
| gridLogIf(ctx, fmt.Errorf("ws write: %w", err)) | ||
| if !xnet.IsNetworkOrHostDown(err, true) { | ||
| gridLogIf(ctx, fmt.Errorf("ws write: %w", err)) | ||
| } | ||
| return | ||
| } |
There was a problem hiding this comment.
Apply valid deadlines and only unexpected logs, not repeated network errors.
harshavardhana
approved these changes
May 17, 2024
harshavardhana
force-pushed
the
pause-container-test
branch
from
23766c5
to
6998367
Compare
harshavardhana
changed the title
harshavardhana
changed the title
harshavardhana
force-pushed
the
pause-container-test
branch
4 times, most recently
from
12a1a10
to
80daf94
Compare
harshavardhana
force-pushed
the
pause-container-test
branch
from
80daf94
to
08e342f
Compare
…ario Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io> Signed-off-by: Harshavardhana <harsha@minio.io>
harshavardhana
force-pushed
the
pause-container-test
branch
from
08e342f
to
b6ced9c
Compare
vadmeste
approved these changes
May 20, 2024
klauspost
reviewed
May 20, 2024
Comment on lines
-618
to
-621
| err = conn.SetWriteDeadline(time.Now().Add(connWriteTimeout)) | ||
| if err != nil { | ||
| return err | ||
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Community Contribution License
All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.
Description
Add sufficient deadlines and countermeasures to handle the hung node scenario
Motivation and Context
This PR tries to address the hung node scenario by adding sufficient
deadlines and counter measures for such an eventuality.
How to test this PR?
This PR already adds the relevant tests; you may reproduce them
locally as needed by following the mint automation piece.
Types of changes
Checklist:
commit-idorPR #here)