Cypress Accessibility+ Add-on

Accessibility automation principles

This guide outlines foundational concepts for using Cypress Accessibility, which also apply to accessibility automation tools in general.

Levels of Accessibility

Cypress Accessibility runs Axe Core® checks on every significant state and variation of the applications you test with Cypress. This kind of automation can catch up to 57% of issues that would appear in a manual audit, but it does not replace the human judgment that is applied during an audit. Auditors test a wide variety of scenarios and technologies in a way that would be impossible to fully automate.

Accessibility can be thought of as three progressive levels:

• Basic Usability: This is the threshold at which a disabled user is likely to be able to make substantial progress through the application, but may still hit major barriers or usability issues. If your contact forms and issue reporting systems are accessible, you will be able to learn about these problems from actual users. Paying attention to issues reported by Cypress and Axe Core® should help you to this point.
• WCAG Conformance: Automated tools like Axe Core® detect many significant accessibility barriers, but human assessment is essential for validating conformance with the Web Content Accessibility Guidelines (WCAG). This is often the standard set by legislation and regulation related to web accessibility. If your goal is regulatory compliance, this is the bar. No generic automation tool can "prove" compliance with this standard, it requires human assessment.
• Good User Experience: This needs to be validated by actual disabled users who are daily users of assistive technology. It's entirely possible to have a website that passes the first two levels, and still offers a poor user experience.

Accessibility is about communicating the nature and structure of the application, and the available actions and information, in a way that allows disabled users to independently understand the interface and complete all tasks. The POUR principles (Perceivable, Operable, Understandable, Robust) form the foundation for effective accessibility beyond specific tests or checks. While Axe Core® tells us about many genuine accessibility barriers, it's important to leverage that information into making good decisions about accessibility and providing a truly equal experience.

False positives

Cypress Accessibility and Axe Core® attempt to minimize false positives, but they can still occur. Some example of false positives include:

• Invalid States: False positives may arise from testing a partially hydrated DOM or other states users would never encounter. You can report invalid states by using the 'Provide Feedback' button in the Cypress UI and we'll continuously improve the product to account for these scenarios.
• Incorrect Violations: Axe Core® may report violations due to bugs or limitations in its rules. Since Axe Core® is an open-source product, if and when we encounter bugs, we can open issues in the project and even make pull requests if needed.

Commonly mistaken as false positives

Some issues that are not false positives, but may seem like them, include:

• Loading States: Legitimate user-facing loading states can present as potential false positives. They inform users about pending data and provide fallback functionality if data is slow to load. It's a good idea in front-end development to "make impossible states impossible", and in doing so you can follow the accessibility principle of Robustness.
• Assistive Technology Support: Certain accessibility techniques are not fully supported by all assistive technologies like screen readers or voice control software. Axe Core® accounts for this. The software and operating systems tested against are listed in their Accessibility Supported document. For example: it can seem like a false positive from Axe Core® if a failing technique used in your code appears to work correctly in a certain screen reader, browser and operating system combination that you test with. But if it does not have sufficient accessibility support in commonly used technology configurations, Axe Core® will intentionally fail that technique, allowing you to switch to something with better support.

Summary

Accessibility automation is a crucial part of an accessibility strategy in any fast-moving, modern codebase. When multiple code changes are merged on a daily or weekly basis, there is no possibility to run manual checks on every state and variation of the application prior to merge. Understanding some of the details of an "always on" set of accessibility checks in your test pipeline, where a lot of states are automatically detected, will help you develop a feel for how Cypress Accessibility behaves and what kinds of things it is best at finding, as well as what might be left over for other forms of testing.