Flaky Test Management
What you'll learn​
- How to detect and manage flaky tests in Cypress Cloud
- How to enable flake detection and alerting
- The benefits of using flaky test management features
By enabling test retries, Cypress Cloud can detect, flag, and track flaky tests from your recorded Cypress test runs in your CI/CD pipeline.
A test is considered to be flaky when it can pass and fail across multiple retry attempts without any code changes. For example, a test is executed and fails, then the test is executed again, without any change to the code, but this time it passes.
Cypress has the ability to automatically retry failed tests to mitigate flaky tests from failing entire test runs or CI builds.
Flake Detection​
Premium Cypress Cloud Feature
Test flake detection is available to users with a Team Cypress Cloud plan.
One way to battle flaky tests is to detect and monitor them as they occur in an organized and methodical manner such that you can assess their severity to assist with prioritizing their fix.
Enable test retries​
Test retries is the fundamental mechanism that enables the detection of flaky tests by Cypress Cloud. Therefore, enabling test retries is required to take advantage of any flaky test management feature.
If you want more control over the pass/fail result of any test deemed flaky by test retries, you may want to use experimental test retries, which is available as of Cypress 13.4.0.
Flagging Flaky Tests​
Test runs with flaky tests will be flagged with the number of flaky of tests within Cypress Cloud "Latest runs" page. Flaky tests runs can also be filtered in and out via the "Flaky" filter within this page.
Any failure across multiple test run attempts triggered by test retrying will result in a given test case to be flagged as flaky.
Flaky Test Analytics​
Premium Cypress Cloud Feature
Test flake analytics are available to users with a Team Cypress Cloud plan.
The flaky tests analytics page provides a birds-eye-view on the state of flake within your project by showing:
- Plot of the number of flaky tests over time.
- Overall flakiness level of the entire project.
- Number of flake tests grouped by their severity.
- Filterable log of all flaky test cases ordered by severity.
Test flake severity is determined by the frequency of flake or the flake rate of a given test. Flake severity level can be used to prioritize the work needed to resolve flake issues.
Selecting any of the flaky test cases in the analytics page will reveal a test case details panel that shows:
- Historical log of latest flaky runs
- Most common errors across the runs of the test case
- Related test case changelog
- Plot of failure rate and flaky rate over time.
All these test case level details provide deeper context around the occurrences of flake over time to assist with debugging the root cause.
Failure Rate vs Flake Rate​
It is important to understand the distinction between failure rate and flake rate when test retries is enabled. These two metrics are tracked over time for each flaky test case within the flaky tests analytics page:
A test case flagged as flaky could have still passed after multiple test retry attempts. The test result status of individual test retry attempts is separate and distinct from the final test status.
For example, a project is configured to retry failing tests up to 3 times. The first two attempts fail, but the last and third attempt passes, resulting in a final status of passing.
With this concept in mind, it is possible to always have zero final failure rate while exhibiting flake as demonstrated below:
Flake Alerting​
Premium Cypress Cloud Feature
Test flake alerting is available to users with a Team Cypress Cloud plan.
Cypress Cloud can provide alerts via source control (GitHub, GitLab, BitBucket), Slack, and Microsoft Teams integrations to further assist with staying on top of flake occurrences.
GitHub​
Flake alerting via GitHub PR comments and status checks can be enabled within a project's GitHub integration settings:
After enabling GitHub flake alerting, GitHub PR comments will show the number of flaky tests associated with the PR within the test summary, and include a "Flakiness" section highlighting the specific tests that flaked.
Slack​
Flake alerting via Slack can be enabled within Slack integration settings:
After enabling Slack alerts, Cypress Cloud will send Slack messages whenever flaky tests are detected:
See Also​
- Read our blog posts about fighting the test flake.