chore: retry all tests sans some exclusions#8995
Merged
basvandijk merged 3 commits intomasterfrom Feb 23, 2026
Merged
Conversation
cgundy
approved these changes
Feb 23, 2026
basvandijk
added a commit
that referenced
this pull request
Feb 23, 2026
basvandijk
deleted the
basvandijk/retry-all-tests-except-for-exclusions
branch
pietrodimarco-dfinity
pushed a commit
that referenced
this pull request
Feb 28, 2026
Flaky tests should be detected ASAP such that they can be fixed ASAP. At the moment only system-tests (`//rs/tests/.*`) and some specifically flagged tests (`flaky=True`) are automatically retried on failure (up to 3 times) to detect if they're flaky. We recently had some tests that were failing or timing out often that we expect are actually flaky. So we had to flag them with `flaky=True` such that after some period we have gathered enough data to know whether they're flaky or not. This process causes manual labour, is time consuming and prevents us from fixing flakiness ASAP. So this commit moves `--flaky_test_attempts` from a whitelist approach, where only a selected number of tests where automatically retried, to a blacklist approach, where all tests are retried except for the `//pre-commit/...` tests since they have never been flaky (and we intend to move them out of `bazel test`). After running with this for a while we'll remove all [`flaky = True`](https://sourcegraph.com/search?q=repo:%5Egithub%5C.com/dfinity/ic%24+%22flaky+%3D+True%22&patternType=keyword&sm=0) settings from our targets.
pietrodimarco-dfinity
pushed a commit
that referenced
this pull request
Feb 28, 2026
Flaky tests should be detected ASAP such that they can be fixed ASAP. At the moment only system-tests (`//rs/tests/.*`) and some specifically flagged tests (`flaky=True`) are automatically retried on failure (up to 3 times) to detect if they're flaky. We recently had some tests that were failing or timing out often that we expect are actually flaky. So we had to flag them with `flaky=True` such that after some period we have gathered enough data to know whether they're flaky or not. This process causes manual labour, is time consuming and prevents us from fixing flakiness ASAP. So this commit moves `--flaky_test_attempts` from a whitelist approach, where only a selected number of tests where automatically retried, to a blacklist approach, where all tests are retried except for the `//pre-commit/...` tests since they have never been flaky (and we intend to move them out of `bazel test`). After running with this for a while we'll remove all [`flaky = True`](https://sourcegraph.com/search?q=repo:%5Egithub%5C.com/dfinity/ic%24+%22flaky+%3D+True%22&patternType=keyword&sm=0) settings from our targets.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flaky tests should be detected ASAP such that they can be fixed ASAP.
At the moment only system-tests (
//rs/tests/.*) and some specifically flagged tests (flaky=True) are automatically retried on failure (up to 3 times) to detect if they're flaky.We recently had some tests that were failing or timing out often that we expect are actually flaky. So we had to flag them with
flaky=Truesuch that after some period we have gathered enough data to know whether they're flaky or not. This process causes manual labour, is time consuming and prevents us from fixing flakiness ASAP.So this commit moves
--flaky_test_attemptsfrom a whitelist approach, where only a selected number of tests where automatically retried, to a blacklist approach, where all tests are retried except for the//pre-commit/...tests since they have never been flaky (and we intend to move them out ofbazel test).After running with this for a while we'll remove all
flaky = Truesettings from our targets.