Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(stability): add target to focus and repeat new or modified tests #6418

Draft
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

slonka
Copy link
Contributor

@slonka slonka commented Mar 31, 2023

Added a new target to focus and repeat tests. I hope that this will help us preventing introducing flakes in the first place.

How to tests this change:

  1. checkout this branch
  2. create a new branch, add a tests (e.g. copy "should allow the traffic with default traffic permission" from universal tests and call it "should allow the traffic with default traffic permission 2", this is how I tested it)
  3. run make TEST_LIST_BRANCH_TO_DIFF_AGAINST=focus-and-repeat-tests test/e2e/prevent-flakes #where focus-and-repeat-tests is the branch checked out in 1
  4. the output should contain "This was attempt x of 10."

The next step would be to put this in circleci config and run alongside e2e tests, but for this to work the target needs to be in master (or TEST_LIST_BRANCH_TO_DIFF_AGAINST) branch.

In the future this could be extended to include tests that contents have changed (by using git diff, line number and matching that to the nearest tests) or even changed functionality by checking the dependency tree.

Checklist prior to review

  • Link to relevant issue as well as docs and UI issues --
  • This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
  • Tests (Unit test, E2E tests, manual test on universal and k8s) --
  • Do you need to update UPGRADE.md? --
  • Does it need to be backported according to the backporting policy? --
  • Do you need to explicitly set a > Changelog: entry here or add a ci/ label to run fewer/more tests?

Signed-off-by: slonka <slonka@users.noreply.github.com>
…tests

Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
@slonka slonka added the ci/skip-test PR: Don't run unit and e2e tests (maybe this is just a doc change) label Mar 31, 2023
@bartsmykla
Copy link
Contributor

@slonka what is the status of this PR? How much work is still needed to finish it?

@slonka
Copy link
Contributor Author

slonka commented Jun 21, 2023

Hey, it was pretty much in a working state. Maybe we could talk about this during one of the team meetings.

@slonka
Copy link
Contributor Author

slonka commented Oct 30, 2023

closing as people did not ask for this nor did I use this, might reopen in the future

@slonka slonka closed this Oct 30, 2023
@slonka slonka reopened this Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci/skip-test PR: Don't run unit and e2e tests (maybe this is just a doc change)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants