{ "doc": { "id": "accessibility/block-pull-requests", "title": "Block pull requests and set policies | Cypress Accessibility Documentation", "description": "Set policies and block pull requests automatically with Cypress Accessibility's Results API, enabling custom CI workflows to enforce accessibility standards and prevent regressions.", "section": "accessibility", "source_path": "/llm/markdown/accessibility/block-pull-requests.md", "version": "998ed2fab5510b3f672383c6cb23238bf481edb8", "updated_at": "2026-05-24T20:45:33.543Z", "headings": [ { "id": "accessibility/block-pull-requests#block-pull-requests-and-set-policies", "text": "Block pull requests and set policies", "level": 1 }, { "id": "accessibility/block-pull-requests#using-the-results-api", "text": "Using the Results API", "level": 2 }, { "id": "accessibility/block-pull-requests#implementing-a-status-check", "text": "Implementing a status check", "level": 2 }, { "id": "accessibility/block-pull-requests#defining-policies-in-the-verification-step", "text": "Defining policies in the verification step", "level": 2 }, { "id": "accessibility/block-pull-requests#using-profiles-for-pr-specific-configuration", "text": "Using Profiles for PR-specific configuration", "level": 2 }, { "id": "accessibility/block-pull-requests#comparing-against-a-baseline", "text": "Comparing against a baseline", "level": 2 }, { "id": "accessibility/block-pull-requests#baseline-structure", "text": "Baseline structure", "level": 3 }, { "id": "accessibility/block-pull-requests#complete-example", "text": "Complete example", "level": 3 }, { "id": "accessibility/block-pull-requests#key-concepts-in-the-example-script", "text": "Key concepts in the example script", "level": 3 }, { "id": "accessibility/block-pull-requests#new-failing-rules", "text": "New failing rules", "level": 4 }, { "id": "accessibility/block-pull-requests#resolved-rules", "text": "Resolved rules", "level": 4 }, { "id": "accessibility/block-pull-requests#severity-filtering", "text": "Severity filtering", "level": 4 }, { "id": "accessibility/block-pull-requests#view-level-comparison", "text": "View-level comparison", "level": 4 }, { "id": "accessibility/block-pull-requests#best-practices", "text": "Best practices", "level": 3 }, { "id": "accessibility/block-pull-requests#when-to-update-the-baseline", "text": "When to update the baseline", "level": 4 }, { "id": "accessibility/block-pull-requests#handling-partial-reports", "text": "Handling partial reports", "level": 4 }, { "id": "accessibility/block-pull-requests#managing-baseline-across-branches", "text": "Managing baseline across branches", "level": 4 }, { "id": "accessibility/block-pull-requests#storing-the-baseline", "text": "Storing the baseline", "level": 4 } ] }, "content": { "type": "root", "children": [ { "type": "heading", "depth": 1, "children": [ { "type": "text", "value": "Block pull requests and set policies" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Cypress Accessibility reports are generated server-side in Cypress Cloud, based on test artifacts uploaded during execution. This ensures there is no performance impact on your Cypress test runs, but also means that nothing in your Cypress pipeline will fail due to accessibility issues that are detected. Failing a build is a fully opt-in step based on your handling of the results in your CI process." } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Using the Results API" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/guides/results-api.md", "children": [ { "type": "text", "value": "Cypress Accessibility Results API" } ] }, { "type": "text", "value": " allows you to access accessibility results post-test run, enabling workflows like blocking pull requests or triggering alerts based on specific accessibility criteria. This involves adding a dedicated accessibility verification step to your CI pipeline. With a Cypress helper function, you can automatically fetch the report for the relevant test run within the CI build context. Below is an example of how to block a pull request in GitHub Actions if accessibility violations occur:" } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Implementing a status check" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The Results API offers full flexibility to analyze results and take tailored actions. It can also integrate with status checks on pull requests. Below is an example of a passing accessibility verification step:" } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Defining policies in the verification step" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/guides/results-api.md", "children": [ { "type": "text", "value": "Results API Documentation" } ] }, { "type": "text", "value": " provides detailed guidance on the API's capabilities. Here's a simplified example demonstrating how to enforce a \"no new failing accessibility rules\" policy:" } ] }, { "type": "code", "lang": null, "meta": null, "value": "const { getAccessibilityResults } = require('@cypress/extract-cloud-results')// List of known failing rulesconst rulesWithExistingViolations = [ 'button-name', // Add more rules as needed]// Fetch accessibility resultsgetAccessibilityResults().then((results) => { // Identify new rule violations const newRuleViolations = results.rules.filter((rule) => { return !rulesWithExistingViolations.includes(rule.name) }) if (newRuleViolations.length > 0) { // Trigger appropriate action if new violations are detected throw new Error( `${newRuleViolations.length} new rule violations detected. These must be resolved.` ) }})" }, { "type": "paragraph", "children": [ { "type": "text", "value": "By examining the results and customizing your response, you gain maximum control over how to handle accessibility violations. Leverage CI environment context, such as tags, to fine-tune responses to specific accessibility outcomes." } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Using Profiles for PR-specific configuration" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "You can use " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/configuration/profiles.md", "children": [ { "type": "text", "value": "Profiles" } ] }, { "type": "text", "value": " to apply different configuration settings for pull request runs versus regression runs. This allows you to:" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Use a narrow, focused configuration for PR runs that blocks merges based on critical accessibility issues" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Maintain a broader configuration for regression runs that tracks all issues for long-term monitoring" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Apply team-specific configurations when multiple teams share the same Cypress Cloud project" } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "For example, you might configure a profile named `aq-config-pr` that excludes non-critical pages and focuses only on the most important accessibility rules, while your base configuration includes all pages and rules for comprehensive regression tracking." } ] }, { "type": "code", "lang": null, "meta": null, "value": "cypress run --record --tag \"aq-config-pr\"" }, { "type": "paragraph", "children": [ { "type": "text", "value": "This approach ensures that PR checks are fast and focused, while still maintaining comprehensive reporting for your full test suite." } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Comparing against a baseline" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Comparing current results against a stored baseline allows you to detect only new violations that have been introduced, while ignoring existing known issues. This approach is more sophisticated than simply maintaining a list of known failing rules, as it tracks violations at the view level and can detect both regressions (new violations) and improvements (resolved violations)." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "This is particularly useful in CI/CD pipelines where you want to fail builds only when new accessibility violations are introduced, allowing you to address existing issues incrementally without blocking deployments." } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Baseline structure" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "You can use any format you like as baseline for comparing reports. In the example code below we generate a baseline in a simplified format, which captures the state of accessibility violations from a specific run. The Results API handler code logs this for every run so that it can be easily copied as a new reference point." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "It includes:" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "runNumber: The run number used as the baseline reference" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "severityLevels: The severity levels to track (e.g., `['critical', 'serious', 'moderate', 'minor']`)" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "views: An object mapping view display names to arrays of failed rule names for that view" } ] } ] } ] }, { "type": "code", "lang": null, "meta": null, "value": "{ \"runNumber\": \"111\", \"severityLevels\": [ \"critical\", \"serious\", \"moderate\", \"minor\" ], \"views\": { \"/\": [ \"aria-dialog-name\", \"heading-order\", \"scrollable-region-focusable\" ], \"/authorizations\": [ \"heading-order\", \"listitem\", \"region\" ] }}" }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Complete example" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The following example demonstrates how to compare current results against a baseline, detect new violations, identify resolved violations, and generate a new baseline when changes are detected." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "scripts/compareAccessibilityBaseline.js" } ] }, { "type": "code", "lang": null, "meta": null, "value": "require('dotenv').config()const { getAccessibilityResults } = require('@cypress/extract-cloud-results')const fs = require('fs')const TARGET_SEVERITY_LEVELS = ['critical', 'serious', 'moderate', 'minor']// Parse the run number from an accessibility report URLconst parseRunNumber = (url) => { return url.split('runs/')[1].split('/accessibility')[0]}// Define your baseline - beginning with no details// will list all current violations as \"new\", and log a// baseline that you can copy and save as your starting pointconst baseline = { runNumber: '', severityLevels: ['critical', 'serious', 'moderate', 'minor'], views: {},}getAccessibilityResults().then((results) => { // Create objects to store the results const viewRules = {} const viewsWithNewFailedRules = {} const viewsWithMissingRules = {} // Iterate through each view in current results results.views.forEach((view) => { const displayName = view.displayName const ruleNames = view.rules.map((rule) => rule.name) // Add to our results object viewRules[displayName] = ruleNames // Check for new failing rules if (view.rules.length) { view.rules.forEach((rule) => { if (!TARGET_SEVERITY_LEVELS.includes(rule.severity)) { return } // If the view exists in baseline and the rule is not in baseline's failed rules if (!baseline.views?.[displayName]?.includes(rule.name)) { if (viewsWithNewFailedRules[displayName]) { viewsWithNewFailedRules[displayName].newFailedRules.push({ name: rule.name, url: rule.accessibilityReportUrl, }) } else { viewsWithNewFailedRules[displayName] = { newFailedRules: [ { name: rule.name, url: rule.accessibilityReportUrl, }, ], } } } }) } }) // Check for rules in baseline that are not in current results (resolved rules) Object.entries(baseline.views).forEach(([displayName, baselineRules]) => { const currentRules = viewRules[displayName] || [] const resolvedRules = baselineRules.filter( (rule) => !currentRules.includes(rule) ) if (resolvedRules.length > 0) { viewsWithMissingRules[displayName] = { resolvedRules } } }) // Report any missing rules const countOfViewsWithMissingRules = Object.keys(viewsWithMissingRules).length const countOfViewsWithNewFailedRules = Object.keys( viewsWithNewFailedRules ).length if (countOfViewsWithMissingRules || countOfViewsWithNewFailedRules) { // Generate and log the new baseline values if there has been a change const newBaseline = generateBaseline(results) console.log('\\nTo use this run as the new baseline, copy these values:') console.log(JSON.stringify(newBaseline, null, 2)) fs.writeFileSync('new-baseline.json', JSON.stringify(newBaseline, null, 2)) } if (countOfViewsWithMissingRules) { console.log( '\\nThe following Views had rules in the baseline that are no longer failing. This may be due to improvements in these rules, or because you did not run as many tests in this run as the baseline run:' ) console.dir(viewsWithMissingRules, { depth: 3 }) } else if (!countOfViewsWithNewFailedRules) { console.log( '\\nNo new or resolved rules were detected. All violations match the baseline.\\n' ) } if (countOfViewsWithNewFailedRules) { // Report any new failing rules console.error( '\\nThe following Views had rules violated that were previously passing:' ) console.dir(viewsWithNewFailedRules, { depth: 3 }) throw new Error( `${countOfViewsWithNewFailedRules} Views contained new failing accessibility rules.` ) } return viewRules})function generateBaseline(results) { try { // Create an object to store the results const viewRules = {} // Iterate through each view results.views.forEach((view) => { // Get the displayName and extract the rule names const displayName = view.displayName const ruleNames = view.rules .filter((rule) => TARGET_SEVERITY_LEVELS.includes(rule.severity)) .map((rule) => rule.name) // Add to our results object viewRules[displayName] = ruleNames }) const runNumber = parseRunNumber(results.views[0].accessibilityReportUrl) return { runNumber, severityLevels: TARGET_SEVERITY_LEVELS, views: viewRules, } } catch (error) { console.error('Error parsing accessibility results:', error) return null }}" }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Key concepts in the example script" } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "New failing rules" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "A new failing rule is a rule that has violations on a View in the current run that were not present in the baseline. These represent regressions that need to be addressed. The script will fail the build if any new failing rules are detected." } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "Resolved rules" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "A resolved rule is a rule that was present in the baseline but no longer has violations in the current run. These represent improvements in accessibility. The script reports these but does not fail the build, allowing you to track progress, and printing out an updated baseline to copy over when things are fixed." } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "Severity filtering" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The baseline comparison only tracks rules at the severity levels you specify in `TARGET_SEVERITY_LEVELS`. This allows you to focus on the severity levels that matter most to your team. Rules at other severity levels are ignored during comparison." } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "View-level comparison" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Violations are tracked per view (URL pattern or component), allowing you to see exactly which pages or components have regressed or improved. This granular tracking makes it easier to identify the source of changes and assign fixes to the appropriate teams." } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Best practices" } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "When to update the baseline" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Update your baseline when:" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "You've fixed accessibility violations and want to prevent regressions" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "You've accepted certain violations as known issues that won't block deployments" } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "We recommend storing the baseline in version control so it's versioned alongside your code and accessible in CI environments, but some users prefer updating the baseline automatically based on the newest results from a full passing test run on the main branch of the project. This means any PRs will be compared against the reference run that makes sense at the time the results are analyzed, and avoids needing to manually own the standard." } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "Handling partial reports" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "If a run is cancelled or incomplete, the Results API may return a partial report. Consider checking `summary.isPartialReport` before comparing against the baseline, as partial reports may not include all views and could produce false positives." } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "Managing baseline across branches" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "You may want different baselines for different branches (e.g., `main` vs feature branches). Consider storing baselines in branch-specific files or using environment variables to specify which baseline to use." } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "Storing the baseline" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Common approaches for storing baselines:" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Version control: Commit the baseline JSON file to your repository" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "CI artifacts: Store baselines as build artifacts that can be retrieved in subsequent runs" } ] } ] } ] } ] }, "token_estimate": 2145 }