Skip to main content
Cypress AccessibilityPremium Solution

Block pull requests and set policies

Cypress Accessibility reports are generated server-side in Cypress Cloud, based on test artifacts uploaded during execution. This ensures there is no performance impact on your Cypress test runs, but also means that nothing in your Cypress pipeline will fail due to accessibility issues that are detected. Failing a build is a fully opt-in step based on your handling of the results in your CI process.

Using the Results API​

The Cypress Accessibility Results API allows you to access accessibility results post-test run, enabling workflows like blocking pull requests or triggering alerts based on specific accessibility criteria. This involves adding a dedicated accessibility verification step to your CI pipeline. With a Cypress helper function, you can automatically fetch the report for the relevant test run within the CI build context. Below is an example of how to block a pull request in GitHub Actions if accessibility violations occur:

On the left hand side, a list of GitHub Actions jobs is presented related to building, testing and deploying a website. Everything has a green checkmark except the `verify-accessibility-results` job, and the 'test' job that is its parents. On the right hand side we see the terminal output for that job which warns us that the 'button-name' rule has been broken, and that it was previously passing.

Implementing a status check​

The Results API offers full flexibility to analyze results and take tailored actions. It can also integrate with status checks on pull requests. Below is an example of a passing accessibility verification step:

An example list of GitHub Status Checks including a passing `verify-accessibility-results` check

Defining policies in the verification step​

The Results API Documentation provides detailed guidance on the API's capabilities. Two rule-level policies cover most teams, and they're mirror images of each other. Both read results.rules, which lists only the rules that failed in the run.

PolicyBlocks whenBest for
Allowlist known failuresAny rule fails that isn't on your known-issues listReports that are close to clean, where anything unexpected should stop a merge
Protect what you've fixedA rule you already remediated fails againA large backlog you're working through, where most rules still fail

For a policy that tracks violations per view rather than per rule, see comparing against a baseline.

Policy: block any rule that isn't already failing​

This enforces "no new failing accessibility rules". You keep a list of the rules you already know about, and anything outside it fails the build:

const { getAccessibilityResults } = require('@cypress/extract-cloud-results')

// List of known failing rules
const rulesWithExistingViolations = [
  'button-name',
  // Add more rules as needed
]

// Fetch accessibility results
getAccessibilityResults().then((results) => {
  // Identify new rule violations
  const newRuleViolations = results.rules.filter((rule) => {
    return !rulesWithExistingViolations.includes(rule.name)
  })

  if (newRuleViolations.length > 0) {
    // Trigger appropriate action if new violations are detected
    throw new Error(
      `${newRuleViolations.length} new rule violations detected. These must be resolved.`
    )
  }
})

The list shrinks as you fix things, and every entry you remove raises the standard the build enforces.

Policy: block regressions in rules you've fixed​

This is the inverse, and it suits a project still working through a large backlog. Rather than listing what's broken, you list what you've finished, and only a reappearance of those rules fails the build:

scripts/verifyAccessibilityResults.js
const { getAccessibilityResults } = require('@cypress/extract-cloud-results')

// Rules this team has remediated. Add a rule here the day it reaches zero
// violations, so any reappearance is treated as a regression.
const remediatedRules = ['button-name', 'image-alt', 'label']

getAccessibilityResults().then((results) => {
  const regressions = results.rules.filter((rule) =>
    remediatedRules.includes(rule.name)
  )

  if (regressions.length === 0) {
    console.log(`No regressions in ${remediatedRules.length} remediated rules.`)
    return
  }

  console.error('Previously remediated accessibility rules are failing again:')
  regressions.forEach((rule) => {
    console.error(
      `  - ${rule.name} (${rule.severity}): ${rule.accessibilityReportUrl}`
    )
  })

  throw new Error(
    `${regressions.length} remediated rule(s) regressed and must be fixed before merging.`
  )
})

Here the list grows as you work. It never blocks a merge for a rule you haven't gotten to yet, which keeps the gate credible while the backlog is still large. See Fix accessibility violations for how to work through that backlog one rule at a time.

By examining the results and customizing your response, you gain maximum control over how to handle accessibility violations. Leverage CI environment context, such as tags, to fine-tune responses to specific accessibility outcomes.

Using Profiles for PR-specific configuration​

You can use Profiles to apply different configuration settings for pull request runs versus regression runs. This allows you to:

  • Use a narrow, focused configuration for PR runs that blocks merges based on critical accessibility issues
  • Maintain a broader configuration for regression runs that tracks all issues for long-term monitoring
  • Apply team-specific configurations when multiple teams share the same Cypress Cloud project

For example, a profile named aq-config-pr can narrow pull request reports to a single flow that's under active remediation, while your base configuration keeps every page for regression tracking. The first rule includes the flow and the catch-all * rule excludes everything else, because viewFilters apply in order and the first match wins:

App Quality Config
{
  "profiles": [
    {
      "name": "aq-config-pr",
      "comment": "Pull request runs report only on the checkout flow while we remediate it",
      "config": {
        "viewFilters": [
          {
            "pattern": "/checkout/*",
            "include": true
          },
          {
            "pattern": "*",
            "include": false
          }
        ]
      }
    }
  ]
}

Record pull request runs with the matching tag to select the profile:

cypress run --record --tag "aq-config-pr"

This approach ensures that PR checks are fast and focused, while still maintaining comprehensive reporting for your full test suite. A profile can't change which Axe-Core® rules run, so which rules block a merge stays a decision you make in the verification step above.

Comparing against a baseline​

Comparing current results against a stored baseline allows you to detect only new violations that have been introduced, while ignoring existing known issues. This approach is more sophisticated than simply maintaining a list of known failing rules, as it tracks violations at the view level and can detect both regressions (new violations) and improvements (resolved violations).

This is particularly useful in CI/CD pipelines where you want to fail builds only when new accessibility violations are introduced, allowing you to address existing issues incrementally without blocking deployments.

Baseline structure​

You can use any format you like as baseline for comparing reports. In the example code below we generate a baseline in a simplified format, which captures the state of accessibility violations from a specific run. The Results API handler code logs this for every run so that it can be easily copied as a new reference point.

It includes:

  • runNumber: The run number used as the baseline reference
  • severityLevels: The severity levels to track (e.g., ['critical', 'serious', 'moderate', 'minor'])
  • views: An object mapping view display names to arrays of failed rule names for that view
{
  "runNumber": "111",
  "severityLevels": [
    "critical",
    "serious",
    "moderate",
    "minor"
  ],
  "views": {
    "/": [
      "aria-dialog-name",
      "heading-order",
      "scrollable-region-focusable"
    ],
    "/authorizations": [
      "heading-order",
      "listitem",
      "region"
    ]
  }
}

Complete example​

The following example demonstrates how to compare current results against a baseline, detect new violations, identify resolved violations, and generate a new baseline when changes are detected.

scripts/compareAccessibilityBaseline.js
require('dotenv').config()

const { getAccessibilityResults } = require('@cypress/extract-cloud-results')
const fs = require('fs')

const TARGET_SEVERITY_LEVELS = ['critical', 'serious', 'moderate', 'minor']

// Parse the run number from an accessibility report URL
const parseRunNumber = (url) => {
  return url.split('runs/')[1].split('/accessibility')[0]
}

// Define your baseline - beginning with no details
// will list all current violations as "new", and log a
// baseline that you can copy and save as your starting point

const baseline = {
  runNumber: '',
  severityLevels: ['critical', 'serious', 'moderate', 'minor'],
  views: {},
}

getAccessibilityResults().then((results) => {
  // Create objects to store the results
  const viewRules = {}
  const viewsWithNewFailedRules = {}
  const viewsWithMissingRules = {}

  // Iterate through each view in current results
  results.views.forEach((view) => {
    const displayName = view.displayName
    const ruleNames = view.rules.map((rule) => rule.name)

    // Add to our results object
    viewRules[displayName] = ruleNames

    // Check for new failing rules
    if (view.rules.length) {
      view.rules.forEach((rule) => {
        if (!TARGET_SEVERITY_LEVELS.includes(rule.severity)) {
          return
        }

        // If the view exists in baseline and the rule is not in baseline's failed rules
        if (!baseline.views?.[displayName]?.includes(rule.name)) {
          if (viewsWithNewFailedRules[displayName]) {
            viewsWithNewFailedRules[displayName].newFailedRules.push({
              name: rule.name,
              url: rule.accessibilityReportUrl,
            })
          } else {
            viewsWithNewFailedRules[displayName] = {
              newFailedRules: [
                {
                  name: rule.name,
                  url: rule.accessibilityReportUrl,
                },
              ],
            }
          }
        }
      })
    }
  })

  // Check for rules in baseline that are not in current results (resolved rules)
  Object.entries(baseline.views).forEach(([displayName, baselineRules]) => {
    const currentRules = viewRules[displayName] || []
    const resolvedRules = baselineRules.filter(
      (rule) => !currentRules.includes(rule)
    )

    if (resolvedRules.length > 0) {
      viewsWithMissingRules[displayName] = { resolvedRules }
    }
  })

  // Report any missing rules
  const countOfViewsWithMissingRules = Object.keys(viewsWithMissingRules).length
  const countOfViewsWithNewFailedRules = Object.keys(
    viewsWithNewFailedRules
  ).length

  if (countOfViewsWithMissingRules || countOfViewsWithNewFailedRules) {
    // Generate and log the new baseline values if there has been a change
    const newBaseline = generateBaseline(results)
    console.log('\nTo use this run as the new baseline, copy these values:')
    console.log(JSON.stringify(newBaseline, null, 2))
    fs.writeFileSync('new-baseline.json', JSON.stringify(newBaseline, null, 2))
  }

  if (countOfViewsWithMissingRules) {
    console.log(
      '\nThe following Views had rules in the baseline that are no longer failing. This may be due to improvements in these rules, or because you did not run as many tests in this run as the baseline run:'
    )
    console.dir(viewsWithMissingRules, { depth: 3 })
  } else if (!countOfViewsWithNewFailedRules) {
    console.log(
      '\nNo new or resolved rules were detected. All violations match the baseline.\n'
    )
  }

  if (countOfViewsWithNewFailedRules) {
    // Report any new failing rules
    console.error(
      '\nThe following Views had rules violated that were previously passing:'
    )
    console.dir(viewsWithNewFailedRules, { depth: 3 })
    throw new Error(
      `${countOfViewsWithNewFailedRules} Views contained new failing accessibility rules.`
    )
  }

  return viewRules
})

function generateBaseline(results) {
  try {
    // Create an object to store the results
    const viewRules = {}

    // Iterate through each view
    results.views.forEach((view) => {
      // Get the displayName and extract the rule names
      const displayName = view.displayName
      const ruleNames = view.rules
        .filter((rule) => TARGET_SEVERITY_LEVELS.includes(rule.severity))
        .map((rule) => rule.name)

      // Add to our results object
      viewRules[displayName] = ruleNames
    })

    const runNumber = parseRunNumber(results.views[0].accessibilityReportUrl)

    return {
      runNumber,
      severityLevels: TARGET_SEVERITY_LEVELS,
      views: viewRules,
    }
  } catch (error) {
    console.error('Error parsing accessibility results:', error)
    return null
  }
}

Key concepts in the example script​

New failing rules​

A new failing rule is a rule that has violations on a View in the current run that were not present in the baseline. These represent regressions that need to be addressed. The script will fail the build if any new failing rules are detected.

Resolved rules​

A resolved rule is a rule that was present in the baseline but no longer has violations in the current run. These represent improvements in accessibility. The script reports these but does not fail the build, allowing you to track progress, and printing out an updated baseline to copy over when things are fixed.

Severity filtering​

The baseline comparison only tracks rules at the severity levels you specify in TARGET_SEVERITY_LEVELS. This allows you to focus on the severity levels that matter most to your team. Rules at other severity levels are ignored during comparison.

View-level comparison​

Violations are tracked per view (URL pattern or component), allowing you to see exactly which pages or components have regressed or improved. This granular tracking makes it easier to identify the source of changes and assign fixes to the appropriate teams.

Best practices​

When to update the baseline​

Update your baseline when:

  • You've fixed accessibility violations and want to prevent regressions
  • You've accepted certain violations as known issues that won't block deployments

We recommend storing the baseline in version control so it's versioned alongside your code and accessible in CI environments, but some users prefer updating the baseline automatically based on the newest results from a full passing test run on the main branch of the project. This means any PRs will be compared against the reference run that makes sense at the time the results are analyzed, and avoids needing to manually own the standard.

Handling partial reports​

If a run is cancelled or incomplete, the Results API may return a partial report. Consider checking summary.isPartialReport before comparing against the baseline, as partial reports may not include all views and could produce false positives.

Managing baseline across branches​

You may want different baselines for different branches (e.g., main vs feature branches). Consider storing baselines in branch-specific files or using environment variables to specify which baseline to use.

Storing the baseline​

Common approaches for storing baselines:

  • Version control: Commit the baseline JSON file to your repository
  • CI artifacts: Store baselines as build artifacts that can be retrieved in subsequent runs

Contents