Skip to main content
Cypress Accessibility+ Add-on

Compare reports

Accessibility reports from different runs can be compared in the Branch Review area of Cypress Cloud. This allows you to instantly see if any new issues have been introduced, and drill in to see full-page HTML and CSS snapshots showing only the new issues.

See the video below for an example of using Branch Review in a Pull Request:

Use cases​

Comparing the results from different runs is useful in multiple scenarios.

Key use cases:

  • Pre-merge checks: Know if any net-new issues are introduced by UI code changes.
  • Monitoring changes: Compare nightly monitoring runs and track down the introduction of new problems caused by underlying changes in the application.
  • Detecting content issues: Sometimes content editors can introduce accessibility issues unrelated to code changes. Seeing the example issues presented visually, in context, helps you quickly triage whether you are dealing with a recent code change issue, or a content authorship problem.
  • Reviewing AI-generated code changes: The increased use of AI to generate and/or review front-end code creates some increased risks of accessibility regressions making it to production. The increase or decrease of accessibility issues when reviewing a pull request helps you understand the impact of the change.
  • Tracing the introduction of issues: With dropdowns for each run, it's easy to rapidly compare different A and B runs to find the exact commit that introduced a problem.
  • Demonstrating the resolution of issues: Confirm the effect of your improvements, and share overview with your team to more quickly review code changes.

Content of the report​

Cypress Accessibility Branch Review with arrows calling out the sections for new failed rules, resolved rules, and failed rules with changes. New failed rules are defined as rules that just started failing on the new branch. Resolved rules are defined as rules that got fully fixed in the new branch. Failed rules with changes are defined as rules that failed on both branches, but with different element counts or different related pages.

The Branch Review report is prioritized into three sections:

  • New failed rules: Rules that were passing every time they ran on the base run but now have failures in the changed run.
  • Resolved rules: Rules that had some failures detected in the base run but have no failures on the changed run. This helps you celebrate the wins and recognize when a new run has gone green.
  • Failed rules with changes: Rules that were failing in both runs, where the elements with failures detected have increased, decreased, or changed in some way.

This organization of the report brings the most significant results to the top - regressions of rules that had been fully passing in the base run. Increases or decreases in the element counts for rules that already have some failures provide a good sense of progress, but can tend to be noisier, especially if the runs have different content or other conditions have changed in between.

How to compare runs​

The first step is to get to the Branch Review area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed. We refer to be baseline fun for comparison as the base run, and the changes we are comparing with as the changed run.

There are a number of ways to get to Branch Review depending on where you are looking at your Cypress results. In all cases, once you've picked a changed run, you can adjust the automatically-selected base run to any branch or run in the project.

From a Pull Request​

A pull request comment from Cypress with an arrow pointing to the 'View all changes introduced in this branch' link. Test results and some accessibility score and violation information is also visible in the comment. The project has an 86.86% accessibility score, with 9 failed elements. Failed rule counts are 1 critical, 1 serious, 3 moderate, and 0 minor.

Click the "View all changes introduced in this branch" link at the bottom of the Test Results table. You will enter branch review with the current PR's branch and latest run pre-selected as the "changed" run.

From the run list​

Click the branch name associated with the run. This will take you to Branch Review with that branch's newest run pre-selected as the changed run.

A couple of cards showing various details about Cypress test runs, including pass/fail counts and metadata. An arrow points to the branch name.

From the project list​

A card showing the name of an example project and related branches to choose.

The project card shows three active branches for the project. You can click through to any of these to enter Branch Review with the newest run on that branch pre-selected as the changed run.

From the main navigation​

The main Cypress Cloud navigation showing Latest Runs, Branches, and Analytics menu items. An arrow points to Branches.

When inside of a project, you can select "Branches" in the main navigation to see a full, filterable list of available branches and choose one to set as the changed run.

FAQ​

How do I ensure a good comparison?​

The best subjects to compare are passing runs that ran similar tests on the same set of content. This means that each run visited roughly the same pages and completed the same kinds of workflows. In this situation, any diff in the results is likely the result of changes present in the newer run. This is usually what happens out-of-the box when comparing a pull-request branch with your main branch.

That said, it still possible and valid to compare runs from different points in time with different sets of test results, as long as you bear in mind all the potential sources of difference between the two runs, which you can evaluate for yourself as you explore the results.

What is the purpose of the Beta label?​

This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please hit the feedback button and let us know.

Why do I see some views (pages or components) changing from run-to-run?​

URLs with dynamic slugs in them can appear as "new" pages in some situations. This behavior can be adjusted with View configuration to make sure page names will match across runs by wildcarding parts of the URL as needed.

Contents