Skip to main content

Flaky Test Management

By enabling test retries, Cypress Cloud can detect, flag, and track flaky tests from your recorded Cypress test runs in your CI/CD pipeline.

info
What is a flaky test?

A test is considered to be flaky when it can pass and fail across multiple retry attempts without any code changes.

For example, a test is executed and fails, then the test is executed again, without any change to the code, but this time it passes.

info
What are test retries?

Cypress has the ability to automatically retry failed tests to mitigate flaky tests from failing entire test runs or CI builds.

Flake Detection​

tip

Premium Cypress Cloud Feature

Test flake detection is available to users with a Team Cypress Cloud plan.

One way to battle flaky tests is to detect and monitor them as they occur in an organized and methodical manner such that you can assess their severity to assist with prioritizing their fix.

caution
Test Retries & Flake Detection

Test retries is the fundamental mechanism that enables the detection of flaky tests by Cypress Cloud. Therefore, enabling test retries is required to take advantage of any flaky test management feature.

If you want more control over the pass/fail result of any test deemed flaky by test retries, you may want to use experimental test retries, which is available as of Cypress 13.4.0.

Flagging Flaky Tests​

Test runs with flaky tests will be flagged with the number of flaky of tests within Cypress Cloud "Latest runs" page. Flaky tests runs can also be filtered in and out via the "Flaky" filter within this page.

Flagging flaky tests runs in Cypress Cloud

Any failure across multiple test run attempts triggered by test retrying will result in a given test case to be flagged as flaky.

Flaky Test Analytics​

tip

Premium Cypress Cloud Feature

Test flake analytics are available to users with a Team Cypress Cloud plan.

The flaky tests analytics page provides a birds-eye-view on the state of flake within your project by showing:

  • Plot of the number of flaky tests over time.
  • Overall flakiness level of the entire project.
  • Number of flake tests grouped by their severity.
  • Filterable log of all flaky test cases ordered by severity.
info
Flake Severity

Test flake severity is determined by the frequency of flake or the flake rate of a given test. Flake severity level can be used to prioritize the work needed to resolve flake issues.

Flaky tests analytics

Selecting any of the flaky test cases in the analytics page will reveal a test case details panel that shows:

  • Historical log of latest flaky runs
  • Most common errors across the runs of the test case
  • Related test case changelog
  • Plot of failure rate and flaky rate over time.

All these test case level details provide deeper context around the occurrences of flake over time to assist with debugging the root cause.

Flaky tests analytics details panel

Failure Rate vs Flake Rate​

It is important to understand the distinction between failure rate and flake rate when test retries is enabled. These two metrics are tracked over time for each flaky test case within the flaky tests analytics page:

flake rate vs fail rate

A test case flagged as flaky could have still passed after multiple test retry attempts. The test result status of individual test retry attempts is separate and distinct from the final test status.

For example, a project is configured to retry failing tests up to 3 times. The first two attempts fail, but the last and third attempt passes, resulting in a final status of passing.

With this concept in mind, it is possible to always have zero final failure rate while exhibiting flake as demonstrated below:

flake rate vs fail rate

Flake Alerting​

tip

Premium Cypress Cloud Feature

Test flake alerting is available to users with a Team Cypress Cloud plan.

Cypress Cloud can provide alerts via source control (GitHub, GitLab, BitBucket), Slack, and Microsoft Teams integrations to further assist with staying on top of flake occurrences.

GitHub​

Flake alerting via GitHub PR comments and status checks can be enabled within a project's GitHub integration settings:

GitHub flake alert settings

After enabling GitHub flake alerting, GitHub PR comments will show the number of flaky tests associated with the PR within the test summary, and include a "Flakiness" section highlighting the specific tests that flaked.

GitHub flake alert pr comment

Slack​

Flake alerting via Slack can be enabled within Slack integration settings:

Slack flake alert settings

After enabling Slack alerts, Cypress Cloud will send Slack messages whenever flaky tests are detected:

Slack flake alert

See Also​