reef: qa: add a YAML to ignore MGR_DOWN warning #57565
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RCA showed that it is not the NFS code that lead to the warning since the warning occurred before the test cases started to execute, later on after some discussion with the venky and greg, it was found that there were some clog changes made recently which leads to this warning being added to the clog. Digging more further, it was found that the warning is generated when mgr fail is run when there is no mgr available. The reason for unavailability is when `setup_mgrs()` in class `MgrTestCase` stops the mgr daemons, sometimes the mgr just crashes - `mgr handle_mgr_signal *** Got signal Terminated ***` and after which `mgr fail` (again part of `setup_mgrs()`) is run and the `MGR_DOWN` warning is generated. This warning is only evident in nfs is because this is the only fs suite that makes use of class `MgrTestCase`. To support my analysis, I had ran about eight jobs in teuthology and I could not reproduce this warning. Since this is not harming the NFS test cases execution and the logs do mention that the mgr daemon did get restarted (`INFO:tasks.cephadm.mgr.x:Restarting mgr.x (starting--it wasn't running)...`), it is good to conclude that ignoring this warning is the simplest solution. Fixes: https://tracker.ceph.com/issues/65265 Signed-off-by: Dhairya Parmar <dparmar@redhat.com> (cherry picked from commit 7d954ce)
|
jenkins test api |
|
jenkins test make check |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of #56944
Backport tracker: https://tracker.ceph.com/issues/66061
Parent tracker: https://tracker.ceph.com/issues/65265
RCA showed that it is not the NFS code that lead to the warning since the warning occurred before the test cases started to execute, later on after some discussion with the venky and greg, it was found that there were some clog changes made recently which leads to this warning being added to the clog.
Digging more further, it was found that the warning is generated when mgr fail is run when there is no mgr available. The reason for unavailability is when
setup_mgrs()in classMgrTestCasestops the mgr daemons, sometimes the mgr just crashes -mgr handle_mgr_signal *** Got signal Terminated ***and after whichmgr fail(again part ofsetup_mgrs()) is run and theMGR_DOWNwarning is generated.This warning is only evident in nfs is because this is the only fs suite that makes use of class
MgrTestCase. To support my analysis, I had ran about eight jobs in teuthology and I could not reproduce this warning. Since this is not harming the NFS test cases execution and the logs do mention that the mgr daemon did get restarted (INFO:tasks.cephadm.mgr.x:Restarting mgr.x (starting--it wasn't running)...), it is good to conclude that ignoring this warning is the simplest solution.Fixes: https://tracker.ceph.com/issues/65265
Signed-off-by: Dhairya Parmar dparmar@redhat.com
(cherry picked from commit 7d954ce)
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume toxjenkins test windowsjenkins test rook e2e