{ "doc": { "id": "accessibility/results-api", "title": "Accessibility Results API: enforce standards in CI", "description": "Programmatically fetch Accessibility results in a CI environment and fail your pipeline if your standards are not met.", "section": "accessibility", "source_path": "/llm/markdown/accessibility/results-api.md", "version": "8d4bb7c9d1af182786c46b47bac667515b9e5d39", "updated_at": "2026-08-07T13:17:53.464Z", "headings": [ { "id": "accessibility/results-api#results-api", "text": "Results API", "level": 1 }, { "id": "accessibility/results-api#supported-ci-providers", "text": "Supported CI Providers", "level": 2 }, { "id": "accessibility/results-api#installation", "text": "Installation", "level": 2 }, { "id": "accessibility/results-api#usage", "text": "Usage", "level": 2 }, { "id": "accessibility/results-api#1-get-the-results", "text": "1. Get the Results", "level": 3 }, { "id": "accessibility/results-api#basic-example", "text": "Basic example", "level": 4 }, { "id": "accessibility/results-api#how-to-assert-that-only-known-rules-are-failing-in-the-run", "text": "How to assert that only known rules are failing in the run", "level": 4 }, { "id": "accessibility/results-api#how-to-detect-a-change-in-the-axe-core-version", "text": "How to detect a change in the axe-core version", "level": 4 }, { "id": "accessibility/results-api#getaccessibilityresults-arguments", "text": "getAccessibilityResults Arguments", "level": 3 }, { "id": "accessibility/results-api#result-types", "text": "Result Types", "level": 4 }, { "id": "accessibility/results-api#comparing-against-a-baseline", "text": "Comparing against a baseline", "level": 2 }, { "id": "accessibility/results-api#2-add-to-ci-workflow", "text": "2. Add to CI Workflow", "level": 3 }, { "id": "accessibility/results-api#example-job-workflow-update", "text": "Example Job Workflow Update:", "level": 4 }, { "id": "accessibility/results-api#required-ci-environment-variables", "text": "Required CI environment variables", "level": 2 }, { "id": "accessibility/results-api#prerequisites", "text": "Prerequisites", "level": 3 }, { "id": "accessibility/results-api#environment-variables-by-ci-provider", "text": "Environment variables by CI provider", "level": 3 }, { "id": "accessibility/results-api#local-development-example", "text": "Local development example", "level": 3 } ] }, "content": { "type": "root", "children": [ { "type": "heading", "depth": 1, "children": [ { "type": "text", "value": "Results API" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The `@cypress/extract-cloud-results` module provides the `getAccessibilityResults` utility which enables you to programmatically fetch your run's Accessibility results in a CI environment. It determines the Cypress run created for the given CI workflow and will return the Accessibility results associated with that run. The results will be returned once the Cypress run has finished and the Accessibility report has been processed." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "This allows you to review the results within CI and to determine if the results are acceptable or need to be addressed before code changes can merge. It provides overall accessibility scores and violation details for the runs, as well as page- or component-level feedback. This supports a wide variety of needs related to alerting about failures in specific focus areas of the application or creating fine-grained regression monitoring according to the current standards being met by each page." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Examples and use cases" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "This page focuses on how the Results API works and what kinds of information can be accessed. For examples of how this can be used, see our higher-level guides:" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "The " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/guides/block-pull-requests.md", "children": [ { "type": "text", "value": "pull requests and policies guide" } ] }, { "type": "text", "value": " shows what it looks like to use the Results API to set a policy and fail a pull request." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "The guide for " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/guides/detect-changes.md#Results-API", "children": [ { "type": "text", "value": "detecting and managing changes" } ] }, { "type": "text", "value": " shows some other common use cases." } ] } ] } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Supported CI Providers" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Fetching Accessibility results for a run supports fetching results for the following CI providers. Please see the docs below for information on general setup." } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/app/continuous-integration/overview.md#Azure-Pipelines", "children": [ { "type": "text", "value": "Azure" } ] }, { "type": "text", "value": " (requires Cypress v13.13.1)" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/app/continuous-integration/circleci.md", "children": [ { "type": "text", "value": "CircleCI" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/app/continuous-integration/github-actions.md", "children": [ { "type": "text", "value": "GitHub Actions" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/app/continuous-integration/gitlab-ci.md", "children": [ { "type": "text", "value": "GitLab" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/app/continuous-integration/overview.md#Jenkins", "children": [ { "type": "text", "value": "Jenkins" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/app/continuous-integration/aws-codebuild.md", "children": [ { "type": "text", "value": "AWS CodeBuild" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Drone" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/app/continuous-integration/bitbucket-pipelines.md", "children": [ { "type": "text", "value": "Bitbucket" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Buildkite" } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Please reach out to Cypress Support to request support for a different provider." } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Installation" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Install the `@cypress/extract-cloud-results` module in your install step in CI." } ] }, { "type": "code", "lang": null, "meta": null, "value": "npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz" }, { "type": "paragraph", "children": [ { "type": "text", "value": "Do not check this module in as a dependency. You should install it separately outside of your normal module installation. Use `--force` to get the latest version." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "If you check this in as a dependency, your installation will fail when we update the package." } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Usage" } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "1. Get the Results" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Write a script using the `getAccessibilityResults` utility to retrieve the results and perform one or more assertions to verify if the changes are acceptable. This script will be executed in CI." } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "Basic example" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "This snippet uses the `getAccessibilityResults()` helper to log out the results. It assumes your Project ID and Record Key variable are set. The following should work in any of the supported CI Providers out of the box:" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "scripts/verifyAccessibilityResults.js" } ] }, { "type": "code", "lang": null, "meta": null, "value": "// Assuming these environment variables are set:// CYPRESS_PROJECT_ID=your-id// CYPRESS_RECORD_KEY=your-record-keyconst { getAccessibilityResults } = require('@cypress/extract-cloud-results')getAccessibilityResults().then((results) => { // use `console.dir` instead of `console.log` because the data is nested console.dir(results, { depth: Infinity })})" }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "How to assert that only known rules are failing in the run" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The Cypress App " }, { "type": "link", "title": null, "url": "https://github.com/cypress-io/cypress", "children": [ { "type": "text", "value": "repository" } ] }, { "type": "text", "value": " uses the Results API to ensure no new violations have been introduced. You can reference " }, { "type": "link", "title": null, "url": "https://github.com/cypress-io/cypress/blob/develop/scripts/verify-accessibility-results.js", "children": [ { "type": "text", "value": "this script" } ] }, { "type": "text", "value": " as a real example." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "scripts/verifyAccessibilityResults.js" } ] }, { "type": "code", "lang": null, "meta": null, "value": "const { getAccessibilityResults } = require('@cypress/extract-cloud-results')/** * The list of rules that currently have 1+ elements that have been flagged with * violations within the Cypress Accessibility report that need to be addressed. * * Once the violation is fixed in the Accessibility report, * the fixed rule should be removed from this list. * * View the Accessibility report for the Cypress run in the Cloud * for more details on how to address these failures. */const rulesWithExistingViolations = [ 'aria-required-children', 'empty-heading', 'aria-dialog-name', 'link-in-text-block', 'list',]getAccessibilityResults({ projectId: '...', // optional if set from env recordKey: '...', // optional if set from env runTags: [process.env.RUN_TAGS], // required if recording multiple runs}).then((results) => { const { runNumber, accessibilityReportUrl, summary, rules } = results const { total } = summary.violationCounts console.log( `Received ${summary.isPartialReport ? 'partial' : ''} results for run #${runNumber}.` ) console.log(`See full report at ${accessibilityReportUrl}.`) // write your logic to conditionally fail based on the results if (total === 0) { console.log('No Accessibility violations detected!') return } const { critical, serious, moderate, minor } = summary.violationCounts console.log(`${total} Accessibility violations were detected:`) console.log(` - ${critical} critical`) console.log(` - ${serious} serious`) console.log(` - ${moderate} moderate`) console.log(` - ${minor} minor.`) const newRuleViolations = rules.filter((rule) => { return !rulesWithExistingViolations.includes(rule.name) }) if (newRuleViolations.length > 0) { console.error( 'The following rules were violated that were previously passing:' ) console.error(newRuleViolations) throw new Error( `${newRuleViolations.length} rule regressions were introduced and must be fixed.` ) } if (total < rulesWithExistingViolations.length) { console.warn( `It seems you have resolved ${rulesWithExistingViolations.length - total} rule(s). Remove them from the list of problematic rules so regressions are not introduced.` ) } console.log('No new Accessibility violations detected!')})" }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "How to detect a change in the axe-core version" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Each result includes the `axeVersion` used to generate the report. Cypress updates its internal Axe-Core® version periodically, and a new version can add rules or change how issues are detected. You can pin the version your team has reviewed and get a heads-up when it changes, so you can reconcile any new results before they surprise you. See " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/configuration/axe-core-configuration.md#axe-core-library-version-and-updates", "children": [ { "type": "text", "value": "Axe Core® library version and updates" } ] }, { "type": "text", "value": " for the update policy." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "scripts/checkAxeVersion.js" } ] }, { "type": "code", "lang": null, "meta": null, "value": "const { getAccessibilityResults } = require('@cypress/extract-cloud-results')// The Axe-Core® version your team has last reviewed and calibrated against.// Bump this deliberately, after reconciling a new Axe-Core® release.const EXPECTED_AXE_VERSION = '4.12.1'getAccessibilityResults().then((results) => { if (!results) { console.warn('No Accessibility report was found for this run.') return } const { axeVersion, runNumber, accessibilityReportUrl } = results if (axeVersion === EXPECTED_AXE_VERSION) { console.log( `Accessibility report used the expected Axe-Core® ${axeVersion}.` ) return } console.warn( `Cypress Accessibility used Axe-Core® ${axeVersion} for run #${runNumber}, but this project is calibrated against ${EXPECTED_AXE_VERSION}.` ) console.warn( `Review the release notes at https://github.com/dequelabs/axe-core/releases, reconcile any new results, then update EXPECTED_AXE_VERSION. Report: ${accessibilityReportUrl}` ) // Opt in to a hard failure so the pipeline blocks until the bump is reviewed: if (process.env.FAIL_ON_AXE_VERSION_CHANGE === 'true') { throw new Error( `Axe-Core® version changed from ${EXPECTED_AXE_VERSION} to ${axeVersion}. Review the changes and update the expected version.` ) }})" }, { "type": "paragraph", "children": [ { "type": "text", "value": "Warning by default is usually the right posture, since a version change isn't a failure on its own. Set `FAIL_ON_AXE_VERSION_CHANGE=true` when you want CI to stop until someone reviews the bump." } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "`getAccessibilityResults` Arguments" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "`getAccessibilityResults` uses the following attributes to identify the Cypress run and return the Accessibility results:" } ] }, { "type": "code", "lang": null, "meta": null, "value": "getAccessibilityResults({ // The Cypress project ID. // Optional if the CYPRESS_PROJECT_ID env is set // Can be explicitly passed to override the env var projectId: string // The project's record key. // Optional if the CYPRESS_RECORD_KEY env is set // Can be explicitly passed to override the env var recordKey: string // The run tags associated with the run. // Required IF you are recording multiple Cypress runs from a single CI build. // Pass the run tags you used when recording in each run // See below for more information runTags: string[]})" }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "Result Types" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The Accessibility results for the run are returned as an object containing the following data:" } ] }, { "type": "code", "lang": null, "meta": null, "value": "{ // The run number of the identified build. runNumber: number // The run url for the identified build. runUrl: 'https://cloud.cypress.io/projects/:project_id/runs/:run_number' // The status of the identified build. runStatus: 'passed' | 'failed' | 'errored' | 'timedOut' | 'cancelled' | 'noTests' // The url that deep links into the summarized Accessibility report for the identified build. accessibilityReportUrl: 'https://cloud.cypress.io/[...]' // The axe-core library version used when generating the Accessibility report. // See https://github.com/dequelabs/axe-core. Example: 4.10.0 axeVersion: string summary: { // Indicates whether a complete Accessibility report was generated. // For example, if a run was cancelled and the report expected to run // for 20 specs, but only 10 ran, this would result in a partial report. isPartialReport: boolean // The total detected violations and the breakdown by rule severity. violationCounts: { // The count of unique rules that detected a violation. total: number, // The count of unique critical rules that detected a violation. critical: number, // The count of unique serious rules that detected a violation. serious: number, // The count of unique moderate rules that detected a violation. moderate: number, // The count of unique minor rules that detected a violation. minor: number, }, // The accessibility score for the run as a whole, to two decimal places score: number, // The count of distinct failed elements detected during the run, across all views failedElements: number } // The list of violated rules. rules: [{ // The name of the rule. See https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md. name: string // The likely impact the rule has on a user with a disability. severity: 'critical' | 'serious' | 'moderate' | 'minor' // The status of the rule for the run. status: 'violation' // The url that deep links the report for this specific rule violation. accessibilityReportUrl: 'https://cloud.cypress.io/[...]' }] // This list of views with accessibility violations detected, // and details of failed rules on each view views: [ { // The url that deep links the report for this view, with no rule-preselected accessibilityReportUrl: 'https://cloud.cypress.io/[...]' // The name of the view as it appears in the accessibility report displayName: \"/app/get-started/why-cypress\", // The accessibility score for this particular view score: number // The list of violated rules for this specific view rules: Rule[] } ]}" }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Comparing against a baseline" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "For comprehensive examples of comparing results against a baseline, including complete code examples, baseline structure, and best practices, see the " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/guides/block-pull-requests.md#comparing-against-a-baseline", "children": [ { "type": "text", "value": "Block pull requests and set policies" } ] }, { "type": "text", "value": " guide." } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "2. Add to CI Workflow" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "In your CI workflow that runs your Cypress tests," } ] }, { "type": "list", "ordered": true, "start": 1, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Update your install job to install the `@cypress/extract-cloud-results` module." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Pass in the necessary arguments to `getAccessibilityResults`." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Add a new step to the job that runs your Cypress tests to verify the Accessibility results." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "If you record multiple runs in a single CI build, you must record these runs using the `--tag` parameter and then call `getAccessibilityResults` with the `runTags` argument for each run." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "This is necessary to identify each unique run and return a corresponding set of results. The tags are how each run is uniquely identified. Tags can also be used to activate " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/configuration/profiles.md", "children": [ { "type": "text", "value": "Profiles" } ] }, { "type": "text", "value": " that apply different configuration settings to your runs." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Example" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Let's imagine that within a single CI build you call `cypress run --record` multiple times because you're running one set of tests against a `staging` environment, followed by a `production` environment." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "In this scenario, you pass a different `--tag` to each cypress run" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`cypress run --record --tag staging`" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`cypress run --record --tag production`" } ] } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "When calling `getAccessibilityResults` you would then pass these same tags to get the unique set of results for each run" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`getAccessibilityResults({ runTags: ['staging']})`" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`getAccessibilityResults({ runTags: ['production']})`" } ] } ] } ] } ] } ] }, { "type": "heading", "depth": 4, "children": [ { "type": "text", "value": "Example Job Workflow Update:" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "GitHub Actions" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "GitLab" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Jenkins" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Azure" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "CircleCI" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "AWS CodeBuild" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Drone" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Bitbucket" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Buildkite" } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "test_cypress.yaml" } ] }, { "type": "code", "lang": null, "meta": null, "value": "name: My Workflowon: pushenv: CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}jobs: run-cypress: runs-on: ubuntu-24.04 steps: - name: Checkout uses: actions/checkout@v7 - name: install run: npm install - name: Run run: npx cypress run --record+ - name: Verify Accessibility Results+ run: |+ npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ node ./scripts/verifyAccessibilityResults.js" }, { "type": "paragraph", "children": [ { "type": "text", "value": ".git-ci.yml" } ] }, { "type": "code", "lang": null, "meta": null, "value": "name: Run Cypress Testsimage: node:lateststages: - testrun-cypress: stage: test secrets: CYPRESS_RECORD_KEY: vault: vault/cypressRecordKey script: - npm install - npx cypress run --record+ - npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ - node ./scripts/verifyAccessibilityResults.js" }, { "type": "code", "lang": null, "meta": null, "value": "pipeline { agent { docker { image 'cypress/base:22.15.0' } } environment { CYPRESS_PROJECT_ID: 'xxxx' CYPRESS_RECORD_KEY = credentials('cypress-record-key') } stages { stage('build and test') { steps { sh 'npm ci' sh 'npx cypress run --record' } }+ stage('Verify Accessibility Results') {+ steps {+ sh 'npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz'+ sh 'node ./scripts/verifyAccessibilityResults.js'+ }+ } }}" }, { "type": "code", "lang": null, "meta": null, "value": "jobs: - job: run_tests pool: vmImage: 'ubuntu-latest' steps: - task: NodeTool@0 inputs: versionSpec: '20.x' displayName: 'Install Node.js' - script: npm i displayName: 'Install npm dependencies' - script: npx cypress run --record displayName: 'Run Cypress tests' env: # avoid warnings about terminal TERM: xterm CYPRESS_RECORD_KEY: $(CYPRESS_RECORD_KEY)+ - script: |+ npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ node ./scripts/verifyAccessibilityResults.js+ displayName: 'Verify Accessibility Results'+ env:+ CYPRESS_PROJECT_ID: $(CYPRESS_PROJECT_ID)+ CYPRESS_RECORD_KEY: $(CYPRESS_RECORD_KEY)" }, { "type": "code", "lang": null, "meta": null, "value": "version: 2.1jobs: linux-test: docker: - image: cypress/base:22.15.0 working_directory: ~/repo steps: - checkout - run: npm install - run: npx run cypress:run --record+ - run: npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ - run: node ./scripts/verifyAccessibilityResults.jsworkflows: version: 2 tests: jobs: - run-cypress" }, { "type": "paragraph", "children": [ { "type": "text", "value": "buildspec.yaml" } ] }, { "type": "code", "lang": null, "meta": null, "value": "phases: install: runtime-versions: nodejs: latest commands: # Set COMMIT_INFO variables to send Git specifics to Cypress Cloud when recording # https://docs.cypress.io/app/continuous-integration/overview#Git-information - export COMMIT_INFO_BRANCH=\"$(git rev-parse HEAD | xargs git name-rev | cut -d' ' -f2 | sed 's/remotes\\/origin\\///g')\" - export COMMIT_INFO_MESSAGE=\"$(git log -1 --pretty=%B)\" - export COMMIT_INFO_EMAIL=\"$(git log -1 --pretty=%ae)\" - export COMMIT_INFO_AUTHOR=\"$(git log -1 --pretty=%an)\" - export COMMIT_INFO_SHA=\"$(git log -1 --pretty=%H)\" - export COMMIT_INFO_REMOTE=\"$(git config --get remote.origin.url)\" - npm ci pre_build: commands: - npm run cypress:verify - npm run cypress:info build: commands: - CYPRESS_INTERNAL_ENV=staging CYPRESS_PROJECT_ID=[slug] npx cypress run --record --key [KEY]+ post_build:+ commands:+ - npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ - CYPRESS_INTERNAL_ENV=staging CYPRESS_PROJECT_ID=[slug] CYPRESS_RECORD_KEY=[KEY] node ./scripts/verifyAccessibilityResults.js" }, { "type": "paragraph", "children": [ { "type": "text", "value": ".drone.yaml" } ] }, { "type": "code", "lang": null, "meta": null, "value": " kind: pipeline name: defaultenvironment:CYPRESS_PROJECT_ID: example_project_slugCYPRESS_RECORD_KEY:from_secret: example_record_key_secretsteps:- name: test image: node:latest commands: - npm install - npx cypress run --record* - name: validate* image: node:latest* commands:* - npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz* - node ./scripts/verifyAccessibilityResults.js" }, { "type": "paragraph", "children": [ { "type": "text", "value": "bitbucket-pipelines.yml" } ] }, { "type": "code", "lang": null, "meta": null, "value": "image: node:latestpipelines: default: - step: name: Run Cypress tests script: - npm install - npx cypress run --record+ - npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ - node ./scripts/verifyAccessibilityResults.js" }, { "type": "paragraph", "children": [ { "type": "text", "value": "pipeline.yml" } ] }, { "type": "code", "lang": null, "meta": null, "value": "steps: - label: \"Run Cypress tests\" command: - npm install - npx cypress run --record+ - npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ - node ./scripts/verifyAccessibilityResults.js" }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Required CI environment variables" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The `@cypress/extract-cloud-results` helper detects the correct Cloud run by matching environment variables from where it runs against those present when the run was recorded, which is why it works automatically from the same CI context (as in the examples above)." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Knowing which variables it looks for helps with more complex setups and with local iteration: set them to match what was present in CI to pull a specific run (recorded within the last 7 days) from your machine." } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Prerequisites" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Two prerequisites apply to every provider:" } ] }, { "type": "list", "ordered": true, "start": 1, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Record the Cypress run and run your validation script within the same CI run (the same build, workflow, or pipeline)." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Run the validation script after the run has been recorded, either in a separate job that depends on the recording job (using your provider's dependency option, such as `needs`, `dependsOn`, or `requires`), or serially after the recording step in the same job." } ] } ] } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Environment variables by CI provider" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Each CI provider has a unique combination of components, patterns, and environment variables that must be interpreted by this module. Expand a provider below for its essential environment variables." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "GitHub Actions" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "References: " }, { "type": "link", "title": null, "url": "https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions", "children": [ { "type": "text", "value": "Understanding GitHub Actions" } ] }, { "type": "text", "value": " | " }, { "type": "link", "title": null, "url": "https://docs.github.com/en/actions/learn-github-actions/variables#default-environment-variables", "children": [ { "type": "text", "value": "GitHub Actions default environment variables" } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Essential environment variables" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`GITHUB_ACTIONS` - Presence identifies the environment as a GitHub Actions environment." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`GITHUB_RUN_ID` - Value uniquely identifies a GitHub Actions workflow instance. Value does not change as jobs in the workflow are re-executed." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`GITHUB_RUN_ATTEMPT` - Value identifies the workflow instance's attempt index. Value is incremented each time jobs are re-executed." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "GitLab" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "References: " }, { "type": "link", "title": null, "url": "https://docs.gitlab.com/ee/ci/pipelines/", "children": [ { "type": "text", "value": "GitLab CI/CD pipelines" } ] }, { "type": "text", "value": " | " }, { "type": "link", "title": null, "url": "https://docs.gitlab.com/ee/ci/variables/predefined_variables.html", "children": [ { "type": "text", "value": "GitLab predefined variables" } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Essential environment variables" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`GITLAB_CI` - Presence identifies the environment as a GitLab CI environment" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`CI_PIPELINE_ID` - Value uniquely identifies a GitLab pipeline workflow. This value does not change as jobs in the pipeline are retried." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`CI_JOB_NAME` - Value uniquely identifies a single job name within a pipeline. Ex. `run-e2e`" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`CI_JOB_ID` - Value uniquely identifies an execution instance of a job. This value will change each time a job is executed/re-executed." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Jenkins" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "References: " }, { "type": "link", "title": null, "url": "https://www.jenkins.io/doc/", "children": [ { "type": "text", "value": "Jenkins documentation" } ] }, { "type": "text", "value": " | " }, { "type": "link", "title": null, "url": "https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#using-environment-variables", "children": [ { "type": "text", "value": "Jenkins environment variables" } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Jenkins is heavily customizable through the usage of plugins, which limits the amount of assumptions we can make about available environment variables and overall behavior." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "We have implemented Jenkins support within this module using the broadest set of available default values. For the purposes of this documentation, though, we will discuss terms related to " }, { "type": "link", "title": null, "url": "https://www.jenkins.io/doc/book/pipeline/getting-started/", "children": [ { "type": "text", "value": "Jenkins Pipeline support" } ] }, { "type": "text", "value": "." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Essential environment variables" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`JENKINS_HOME` - Presence identifies the environment as a Jenkins environment" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`BUILD_URL` - Value uniquely identifies a Jenkins job execution, including name and id characteristics." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Azure" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "References: " }, { "type": "link", "title": null, "url": "https://learn.microsoft.com/en-us/azure/devops/pipelines/get-started/key-pipelines-concepts?view=azure-devops", "children": [ { "type": "text", "value": "Azure Pipelines key concepts" } ] }, { "type": "text", "value": " | " }, { "type": "link", "title": null, "url": "https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml", "children": [ { "type": "text", "value": "Azure Pipelines predefined variables" } ] } ] }, { "type": "blockquote", "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Note: Cypress v13.13.1 is the earliest Cypress release that records the environment variables necessary for this module to identify runs in an Azure environment. Previous Cypress versions are not supported in Azure pipelines." } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Essential environment variables" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`TF_BUILD` and `AZURE_HTTP_USER_AGENT` - Combined presence identifies the environment as an Azure pipeline environment." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`SYSTEM_PLANID` - Value uniquely identifies a pipeline run. Value does not change as jobs within the pipeline are retried from failure." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`SYSTEM_JOBID` - Value uniquely identifies a job execution. Value changes each time a job is retried from failure, in conjunction with the `SYSTEM_JOBATTEMPT` being incremented." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`SYSTEM_JOBATTEMPT` - Value identifies the pipelines shared attempt index. Value is incremented when jobs are retried from failure." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "CircleCI" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "References: " }, { "type": "link", "title": null, "url": "https://circleci.com/docs/about-circleci/", "children": [ { "type": "text", "value": "About CircleCI" } ] }, { "type": "text", "value": " | " }, { "type": "link", "title": null, "url": "https://circleci.com/docs/variables/#built-in-environment-variables", "children": [ { "type": "text", "value": "CircleCI built-in environment variables" } ] } ] }, { "type": "blockquote", "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Note: Cypress v13.13.1 is the earliest Cypress release that records the environment variables necessary for this module to identify runs in an CircleCI environment. Previous Cypress versions are not supported in CircleCI pipelines." } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Essential environment variables" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`CIRCLECI` - Presence identifies the environment as a CircleCI environment" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`CIRCLE_PIPELINE_ID` - Value uniquely identifies a CircleCI pipeline, created on push or manually triggered through the UI. This value does not change as workflows within the pipeline are re-executed." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`CIRCLE_WORKFLOW_ID` - Value uniquely identifies an instance of a workflow's execution within a pipeline. This value will be updated upon each workflow execution; in other words, retrying a workflow from failure from the Circle UI will create a new workflow with a new `CIRCLE_WORKFLOW_ID` value available to the jobs executed within it." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`CIRCLE_WORKFLOW_JOB_ID` - Value uniquely identifies an execution instance of a named job within a workflow instance." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "AWS CodeBuild" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "References: " }, { "type": "link", "title": null, "url": "https://docs.aws.amazon.com/codebuild/", "children": [ { "type": "text", "value": "AWS CodeBuild documentation" } ] }, { "type": "text", "value": " | " }, { "type": "link", "title": null, "url": "https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-env-vars.html", "children": [ { "type": "text", "value": "AWS CodeBuild environment variables" } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Essential environment variables" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`CODEBUILD_BUILD_ID` - Presence identifies the environment as an AWS CodeBuild environment. Value uniquely identifies a build." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Drone" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "References: " }, { "type": "link", "title": null, "url": "https://docs.drone.io/pipeline/overview/", "children": [ { "type": "text", "value": "Drone pipeline overview" } ] }, { "type": "text", "value": " | " }, { "type": "link", "title": null, "url": "https://docs.drone.io/pipeline/environment/reference/", "children": [ { "type": "text", "value": "Drone environment reference" } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Essential environment variables" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`DRONE` - Presence identifies the environment as a Drone environment." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`DRONE_BUILD_NUMBER` - Value uniquely identifies a Drone build." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Bitbucket" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "References: " }, { "type": "link", "title": null, "url": "https://support.atlassian.com/bitbucket-cloud/docs/", "children": [ { "type": "text", "value": "Bitbucket Cloud documentation" } ] }, { "type": "text", "value": " | " }, { "type": "link", "title": null, "url": "https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/", "children": [ { "type": "text", "value": "Bitbucket variables and secrets" } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Essential environment variables" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`BITBUCKET_BUILD_NUMBER` - Presence identifies the environment as a Bitbucket environment. Value uniquely identifies a build." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`BITBUCKET_STEP_RUN_NUMBER` - Value indicates a build step execution index and increments when a step is retried. Initial value of 1." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Buildkite" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "References: " }, { "type": "link", "title": null, "url": "https://buildkite.com/docs", "children": [ { "type": "text", "value": "Buildkite documentation" } ] }, { "type": "text", "value": " | " }, { "type": "link", "title": null, "url": "https://buildkite.com/docs/pipelines/environment-variables", "children": [ { "type": "text", "value": "Buildkite environment variables" } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Essential environment variables" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`BUILDKITE` - Presence identifies the environment as a Buildkite environment." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`BUILDKITE_BUILD_ID` - Value uniquely identifies a build. This value does not change across different jobs within the same build." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`BUILDKITE_JOB_ID` - Value uniquely identifies a job execution. Each job in a build has a unique job ID." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`BUILDKITE_RETRY_COUNT` - Value indicates the retry attempt for a job. Default value of 0." } ] } ] } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Local development example" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "To iterate on your verification script and see everything working without putting code into your CI environment, simulate the CI context for a specific Cypress run locally. This can save a lot of time when getting started." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "If you executed a run in GitHub Actions and it was recorded to Cypress Cloud, you would set these 4 environment variables to replicate the context of that run locally and execute your local handler script. This is a great way to iterate on your script and verify everything is working as expected, without having to integrate anything in CI. It's also useful for debugging." } ] }, { "type": "code", "lang": null, "meta": null, "value": "CYPRESS_PROJECT_ID=AAACYPRESS_RECORD_KEY=BBBGITHUB_ACTIONS=trueGITHUB_RUN_ID=111GITHUB_RUN_ATTEMPT=1node verifyResults.js" }, { "type": "paragraph", "children": [ { "type": "text", "value": "The Results API will then look for the Cypress Cloud run that matches this run ID. If there is more than one Cypress Cloud run found for that GitHub Actions Run, you can pass run tags to narrow down to one run's report." } ] } ] }, "token_estimate": 4388 }