Skip to main content
Cypress AccessibilityPremium Solution

Compare reports

Accessibility reports from different runs can be compared in the Branch Review area of Cypress Cloud. This allows you to instantly see if any new issues have been introduced, and drill in to see full-page HTML and CSS snapshots showing only the new issues.

See the video below for an example of using Branch Review in a Pull Request:

Use cases​

Comparing the results from different runs is useful in multiple scenarios.

Key use cases:

  • Pre-merge checks: Know if any net-new issues are introduced by UI code changes.
  • Monitoring changes: Compare nightly monitoring runs and track down the introduction of new problems caused by underlying changes in the application.
  • Detecting content issues: Sometimes content editors can introduce accessibility issues unrelated to code changes. Seeing the example issues presented visually, in context, helps you quickly triage whether you are dealing with a recent code change issue, or a content authorship problem.
  • Reviewing AI-generated code changes: The increased use of AI to generate and/or review front-end code creates some increased risks of accessibility regressions making it to production. The increase or decrease of accessibility issues when reviewing a pull request helps you understand the impact of the change.
  • Tracing the introduction of issues: With dropdowns for each run, it's easy to rapidly compare different A and B runs to find the exact commit that introduced a problem.
  • Demonstrating the resolution of issues: Confirm the effect of your improvements, and share the overview with your team to more quickly review code changes.

Content of the report​

Cypress Accessibility Branch Review with arrows calling out the sections for new failed rules, resolved rules, and failed rules with changes. New failed rules are defined as rules that just started failing on the new branch. Resolved rules are defined as rules that got fully fixed in the new branch. Failed rules with changes are defined as rules that failed on both branches, but with different element counts or different related pages.

The Branch Review report is prioritized into three sections:

  • New failed rules: Rules that were passing every time they ran on the base run but now have failures in the changed run.
  • Resolved rules: Rules that had some failures detected in the base run but have no failures on the changed run. This helps you celebrate the wins and recognize when a new run has gone green.
  • Failed rules with changes: Rules that were failing in both runs, where the elements with failures detected have increased, decreased, or changed in some way.

This organization of the report brings the most significant results to the top - regressions of rules that had been fully passing in the base run. Increases or decreases in the element counts for rules that already have some failures provide a good sense of progress, but can tend to be noisier, especially if the runs have different content or other conditions have changed in between.

How to compare runs​

The first step is to get to the Branch Review area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed. You can access this area by clicking the branch name associated with a run, or in several other ways. Learn more about how to compare runs.

FAQ​

How do I ensure a good comparison?​

The best subjects to compare are passing runs that ran similar tests on the same set of content. This means that each run visited roughly the same pages and completed the same kinds of workflows. In this situation, any diff in the results is likely the result of changes present in the newer run. This is usually what happens out-of-the box when comparing a pull-request branch with your main branch.

That said, it still possible and valid to compare runs from different points in time with different sets of test results, as long as you bear in mind all the potential sources of difference between the two runs, which you can evaluate for yourself as you explore the results.

In order to see unified changes for your entire test suite, you need to group all the tests together under a single Cypress run, for each report. Learn more about this in the Branch Review Best Practices documentation.

What is the purpose of the Beta label?​

This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please use the feedback button and let us know.

Why do I see some views (pages or components) changing from run-to-run?​

URLs with dynamic slugs in them can appear as "new" pages in some situations. This behavior can be adjusted with View configuration to make sure page names will match across runs by wildcarding parts of the URL as needed.

Contents