{ "doc": { "id": "cloud/features/checks", "title": "Run Checks: pass/fail quality gates for Cypress Cloud runs", "description": "Define Run Checks in Cypress Cloud to pass or fail each recorded run against thresholds for failed tests, flaky tests, accessibility scores, and UI Coverage, scoped to the branches and tags you choose.", "section": "cloud", "source_path": "/llm/markdown/cloud/features/checks.md", "version": "4a645eeb4b355bfca7d7bb576e96889b8b571465", "updated_at": "2026-11-05T01:52:34.491Z", "headings": [ { "id": "cloud/features/checks#run-checks", "text": "Run Checks", "level": 1 }, { "id": "cloud/features/checks#early-access-notes", "text": "Early Access notes", "level": 3 }, { "id": "cloud/features/checks#defining-run-checks", "text": "Defining Run Checks", "level": 2 }, { "id": "cloud/features/checks#syntax", "text": "Syntax", "level": 2 }, { "id": "cloud/features/checks#check-options", "text": "Check options", "level": 3 }, { "id": "cloud/features/checks#assertion-options", "text": "Assertion options", "level": 3 }, { "id": "cloud/features/checks#assertion-types", "text": "Assertion types", "level": 2 }, { "id": "cloud/features/checks#narrowing-to-views", "text": "Narrowing to views", "level": 3 }, { "id": "cloud/features/checks#narrowing-to-specs", "text": "Narrowing to specs", "level": 3 }, { "id": "cloud/features/checks#scoping-checks-to-runs", "text": "Scoping checks to runs", "level": 2 }, { "id": "cloud/features/checks#how-checks-are-evaluated", "text": "How checks are evaluated", "level": 2 }, { "id": "cloud/features/checks#check-statuses", "text": "Check statuses", "level": 3 }, { "id": "cloud/features/checks#assertion-statuses", "text": "Assertion statuses", "level": 3 }, { "id": "cloud/features/checks#using-check-results-in-ci", "text": "Using check results in CI", "level": 2 }, { "id": "cloud/features/checks#see-also", "text": "See also", "level": 2 } ] }, "content": { "type": "root", "children": [ { "type": "heading", "depth": 1, "children": [ { "type": "text", "value": "Run Checks" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Run Checks evaluate recorded runs in Cypress Cloud against thresholds you define: failed tests, flaky tests, accessibility scores, UI Coverage, and more." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Run Checks represent your team's opinion about what counts as \"acceptable\" or \"needs attention\" in your project. With customization to target specific specs, tags, branches, or pages for your checks, a huge amount of flexibility is possible to ensure each check produces useful signals that match the specific purpose of that run. This creates easy visibility for potential issues in Cypress Cloud itself, and allows for automatically flagging runs that miss the mark." } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Declarative thresholds: Encode limits on failed tests, flaky tests, accessibility scores, failing accessibility elements, UI Coverage scores, and untested elements, without maintaining parallel threshold logic in each CI pipeline." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Cross-product gates: One check can combine assertions about a run's test results, its " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/get-started/introduction.md", "children": [ { "type": "text", "value": "Cypress Accessibility" } ] }, { "type": "text", "value": " report, and its " }, { "type": "link", "title": null, "url": "/llm/markdown/ui-coverage/get-started/introduction.md", "children": [ { "type": "text", "value": "UI Coverage" } ] }, { "type": "text", "value": " report." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Customized for your tags and branches: Apply a check only to runs on certain branches or with certain " }, { "type": "link", "title": null, "url": "/llm/markdown/app/references/command-line.md#cypress-run-tag-lt-tag-gt", "children": [ { "type": "text", "value": "run tags" } ] }, { "type": "text", "value": ", so a release gate on `main` doesn't fail every feature-branch run. Include and exclude specs or pages to fine tune the results for your specific needs." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Run Checks don't change how your tests run or how reports are generated. They only define how Cypress Cloud evaluates a run's results once they're available. Here's an example of how they are presented:" } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Early Access notes" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Run Checks have launched as Early Access with a gradual rollout starting November 5, 2026. There are no limitations on usage and no breaking changes are expected, but Checks do not yet flow into all the usual places your Cypress test results are populated, as outlined below. If a feature is not active yet that you are interested in, please reach out to your usual Cypress point of contact. Feedback and requests are appreciated." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Check results appear on the run's page, in its Properties tab, in the project's runs list, and through the " }, { "type": "link", "title": null, "url": "#Using-check-results-in-CI", "children": [ { "type": "text", "value": "Results API" } ] }, { "type": "text", "value": " (`getCheckResults()`). Support in " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/integrations/webhooks.md", "children": [ { "type": "text", "value": "Cypress Cloud webhooks" } ] }, { "type": "text", "value": " is planned soon." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Run Checks are not yet included in automated " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/integrations/slack.md", "children": [ { "type": "text", "value": "Slack" } ] }, { "type": "text", "value": " notifications or " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/integrations/github.md#Pull-request-comments", "children": [ { "type": "text", "value": "GitHub" } ] }, { "type": "text", "value": " pull request comments, and they don't push to GitHub status checks automatically. To report them as commit statuses today, follow " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/guides/report-run-checks-github.md", "children": [ { "type": "text", "value": "Report Run Checks as GitHub pull request statuses" } ] }, { "type": "text", "value": "." } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Defining Run Checks" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Run Checks are part of your project's App Quality configuration. The `name`, threshold, and scoping you set shape how results appear in Cypress Cloud and in the " }, { "type": "link", "title": null, "url": "#Using-check-results-in-CI", "children": [ { "type": "text", "value": "Results API" } ] }, { "type": "text", "value": "." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Run Checks don't block CI automatically" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "A check is a pass/fail evaluation of data about the run, and does not change the run's outcome. To fail a job or block a merge, fetch results with " }, { "type": "link", "title": null, "url": "#Using-check-results-in-CI", "children": [ { "type": "text", "value": "`getCheckResults()`" } ] }, { "type": "text", "value": " or " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/guides/report-run-checks-github.md", "children": [ { "type": "text", "value": "report them as GitHub commit statuses" } ] }, { "type": "text", "value": "." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Labels like \"release gate\" and workflows such as " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/guides/quarantine-problematic-specs.md", "children": [ { "type": "text", "value": "quarantining problematic specs" } ] }, { "type": "text", "value": " are patterns you build with configuration and CI. Run Checks don't enforce those workflows on their own." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Here is one example of how config values appear in the UI. Most projects use a simpler set of checks than this:" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Configuration lives in Cypress Cloud, not in your repository. To view or change it for a project:" } ] }, { "type": "list", "ordered": true, "start": 1, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Open your project in Cypress Cloud and go to Project Settings." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Select the App Quality tab, which holds the configuration editor." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Edit the configuration as JSON, then Save. Discard reverts unsaved edits back to the last saved version." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The editor validates your JSON against the schema when you save, so an invalid configuration can't be saved, and it shows how long ago the configuration was last saved." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Add a `checks` array to the root of the configuration:" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "App Quality Config" } ] }, { "type": "code", "lang": null, "meta": null, "value": "{\n \"checks\": [\n {\n \"name\": \"release-gate\",\n \"comment\": \"Merge requirements for main\",\n \"include\": { \"branches\": [\"main\"] },\n \"assertions\": [\n { \"type\": \"run.failedTests.max\", \"value\": 0 },\n { \"type\": \"accessibility.score.min\", \"value\": 90 },\n {\n \"type\": \"uiCoverage.score.min\",\n \"value\": 80,\n \"viewMatch\": \"^(Login|Checkout)\",\n \"comment\": \"Critical flows must stay well covered\"\n }\n ]\n }\n ]\n}" }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Syntax" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "App Quality Config" } ] }, { "type": "code", "lang": null, "meta": null, "value": "{\n \"checks\": [\n {\n \"name\": string,\n \"assertions\": [\n {\n \"type\": string,\n \"value\": number,\n // View metrics only\n \"viewMatch\": string,\n // Test result metrics only\n \"include\": { \"specs\": [string] } | [{ \"specs\": [string] }],\n \"exclude\": { \"specs\": [string] } | [{ \"specs\": [string] }],\n \"comment\": string\n }\n ],\n \"include\": { \"branches\": [string], \"tags\": [string] } | [ /* rules */ ],\n \"exclude\": { \"branches\": [string], \"tags\": [string] } | [ /* rules */ ],\n \"comment\": string\n }\n ]\n}" }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Check options" } ] }, { "type": "table", "align": [ null, null, null ], "children": [ { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "Option" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Required" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Description" } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`name`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Required" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Identifies the check in Cypress Cloud and in check results. Must be non-empty and unique across the `checks` array." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`assertions`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Required" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "A non-empty array of " }, { "type": "link", "title": null, "url": "#Assertion-types", "children": [ { "type": "text", "value": "assertions" } ] }, { "type": "text", "value": ". The check passes only when every assertion passes." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`include`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Optional" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "One " }, { "type": "link", "title": null, "url": "#Scoping-checks-to-runs", "children": [ { "type": "text", "value": "run rule" } ] }, { "type": "text", "value": ", or an array of rules. When set, the check applies only to runs that match at least one rule." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`exclude`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Optional" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "One " }, { "type": "link", "title": null, "url": "#Scoping-checks-to-runs", "children": [ { "type": "text", "value": "run rule" } ] }, { "type": "text", "value": ", or an array of rules. The check doesn't apply to runs that match any rule, even when `include` matches." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`comment`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Optional" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "A note about why the check exists. Comments have no effect on behavior." } ] } ] } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Assertion options" } ] }, { "type": "table", "align": [ null, null, null ], "children": [ { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "Option" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Required" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Description" } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`type`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Required" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The metric and the direction of its threshold. See " }, { "type": "link", "title": null, "url": "#Assertion-types", "children": [ { "type": "text", "value": "Assertion types" } ] }, { "type": "text", "value": "." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`value`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Required" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The threshold. Scores accept a number from `0` to `100`. Counts accept a non-negative integer." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`viewMatch`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Optional" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "For `accessibility.*` and `uiCoverage.*` assertions. A regular expression matched against view names; each matching view is held to the threshold on its own. See " }, { "type": "link", "title": null, "url": "#Narrowing-to-views", "children": [ { "type": "text", "value": "Narrowing to views" } ] }, { "type": "text", "value": "." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`include`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Optional" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "For `run.*` assertions. One spec rule, or an array of spec rules, selecting which specs' results are counted. See " }, { "type": "link", "title": null, "url": "#Narrowing-to-specs", "children": [ { "type": "text", "value": "Narrowing to specs" } ] }, { "type": "text", "value": "." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`exclude`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Optional" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "For `run.*` assertions. One spec rule, or an array of spec rules, removing specs from the count." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`comment`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Optional" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "A note about the assertion. Comments have no effect on behavior." } ] } ] } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Assertion types" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Each assertion type names the data it reads and whether its threshold is a minimum (`.min`) or a maximum (`.max`). A `.min` assertion passes when the measured value is greater than or equal to `value`; a `.max` assertion passes when it's less than or equal to `value`." } ] }, { "type": "table", "align": [ null, null, null, null ], "children": [ { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "Type" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "`value`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Narrow with" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Passes when" } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`run.failedTests.max`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Integer, `0`+" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "`include` / `exclude`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The number of failed tests in the run is at most `value`." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`run.flakyTests.max`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Integer, `0`+" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "`include` / `exclude`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The number of flaky tests in the run is at most `value`. Requires flaky test detection." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`accessibility.score.min`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "`0` to `100`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "`viewMatch`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The run's " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/core-concepts/accessibility-score.md", "children": [ { "type": "text", "value": "accessibility score" } ] }, { "type": "text", "value": " is at least `value`." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`accessibility.failedElements.max`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Integer, `0`+" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "`viewMatch`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The number of elements failing accessibility rules is at most `value`." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`uiCoverage.score.min`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "`0` to `100`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "`viewMatch`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The run's UI Coverage score is at least `value`." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`uiCoverage.untestedElements.max`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Integer, `0`+" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "`viewMatch`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The number of untested elements is at most `value`." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "`run.*` assertions read the run's own test results and are available on every project. `run.flakyTests.max` reads the flaky tests that " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/features/flaky-test-management.md", "children": [ { "type": "text", "value": "Flaky Test Management" } ] }, { "type": "text", "value": " detects, so it's " }, { "type": "link", "title": null, "url": "#Assertion-statuses", "children": [ { "type": "text", "value": "unavailable" } ] }, { "type": "text", "value": " on projects without flake detection." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "`accessibility.*` and `uiCoverage.*` assertions read the run's Cypress Accessibility and UI Coverage reports, and are " }, { "type": "link", "title": null, "url": "#Assertion-statuses", "children": [ { "type": "text", "value": "unavailable" } ] }, { "type": "text", "value": " on runs where that product isn't enabled or processed." } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Narrowing to views" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Without `viewMatch`, a view assertion measures the run as a whole: the run-level score, or the total count across all views." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "With `viewMatch`, the assertion selects the views whose name matches the pattern and holds each of them to the threshold on its own. The assertion fails if any matched view misses the threshold. A `viewMatch` that matches no views passes, because there's nothing to measure." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "`viewMatch` is a regular expression matched against the " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/configuration/views.md", "children": [ { "type": "text", "value": "view" } ] }, { "type": "text", "value": " name, not the URL, and it matches anywhere in the name. Use `^` and `$` to match the start or end of the name." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "App Quality Config" } ] }, { "type": "code", "lang": null, "meta": null, "value": "{\n \"type\": \"accessibility.failedElements.max\",\n \"value\": 0,\n \"viewMatch\": \"Checkout\",\n \"comment\": \"No failing elements on any checkout view\"\n}" }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Narrowing to specs" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "`run.*` assertions count tests across every spec in the run by default. Use `include` and `exclude` on the assertion to count only some specs. Each rule is an object with a non-empty `specs` array of regular expressions, matched anywhere in the spec's file path." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The counts from every matched spec are added together, and the total is held to the threshold once. A spec that ran on more than one machine, such as in multiple " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/features/smart-orchestration/parallelization.md", "children": [ { "type": "text", "value": "run groups" } ] }, { "type": "text", "value": ", contributes the results from each." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "App Quality Config" } ] }, { "type": "code", "lang": null, "meta": null, "value": "{\n \"type\": \"run.failedTests.max\",\n \"value\": 0,\n \"include\": { \"specs\": [\"cypress/e2e/checkout/\"] },\n \"exclude\": { \"specs\": [\"/quarantine/\"] },\n \"comment\": \"Checkout specs must pass, apart from quarantined ones\"\n}" }, { "type": "paragraph", "children": [ { "type": "text", "value": "To keep known-bad specs in the suite while gating CI on everything else, see " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/guides/quarantine-problematic-specs.md", "children": [ { "type": "text", "value": "Quarantine problematic specs with Run Checks" } ] }, { "type": "text", "value": ". That pattern is a use of `include` / `exclude`, not a built-in Cypress Cloud quarantine mode." } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Scoping checks to runs" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "By default, a check applies to every run in the project. Use `include` and `exclude` on the check to apply it only to some runs." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Each rule is an object with one or both of these properties:" } ] }, { "type": "table", "align": [ null, null ], "children": [ { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "Property" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Description" } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`branches`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "A non-empty array of regular expressions matched against the run's Git branch." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`tags`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "A non-empty array of regular expressions matched against the run's " }, { "type": "link", "title": null, "url": "/llm/markdown/app/references/command-line.md#cypress-run-tag-lt-tag-gt", "children": [ { "type": "text", "value": "tags" } ] }, { "type": "text", "value": "." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "A rule must set at least one of `branches` or `tags`. `include` and `exclude` each accept a single rule or an array of rules." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Rules match as follows:" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Within a property, any pattern can match. `{ \"branches\": [\"main\", \"release/.*\"] }` matches a run on `main` or on any `release/` branch." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Within a rule, every property must match. `{ \"branches\": [\"main\"], \"tags\": [\"nightly\"] }` matches only runs on `main` that also carry the `nightly` tag." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "Across an array of rules, any rule can match." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "`exclude` wins over `include`. A run that matches both is excluded." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "A run with no branch never matches a `branches` pattern. A check that includes only certain branches doesn't apply to it, and an `exclude` rule on branches doesn't remove it." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Branch and tag patterns match the whole value, so `main` matches the branch `main` but not `feature/main`. To match part of a value, start the pattern with `^` or end it with `$` to anchor it yourself, or use `.*`. Patterns are regular expressions, not globs: write `release/.*` rather than `release/*`. Matching is case-sensitive." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "App Quality Config" } ] }, { "type": "code", "lang": null, "meta": null, "value": "{\n \"checks\": [\n {\n \"name\": \"nightly-regression\",\n \"include\": [\n { \"branches\": [\"main\"], \"tags\": [\"nightly\"] },\n { \"tags\": [\"release-candidate\"] }\n ],\n \"exclude\": { \"branches\": [\"^dependabot/\"] },\n \"assertions\": [\n { \"type\": \"run.failedTests.max\", \"value\": 0 },\n { \"type\": \"run.flakyTests.max\", \"value\": 5 }\n ]\n }\n ]\n}" }, { "type": "paragraph", "children": [ { "type": "text", "value": "A check whose tag rule hasn't matched yet stays `pending` until the run completes. Any " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/features/smart-orchestration/parallelization.md", "children": [ { "type": "text", "value": "run group" } ] }, { "type": "text", "value": " can add tags while the run is in progress, so Cypress Cloud can only rule out a tag rule once every group has reported." } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "How checks are evaluated" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Cypress Cloud starts evaluating checks after the run completes. `run.*` assertions can settle immediately from the run's final test counts. `accessibility.*` and `uiCoverage.*` assertions settle independently as each report finishes processing. Results update in Cypress Cloud while evaluation is in progress." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "The check title, threshold pill, and any spec scope line come from your config. Product labels such as Accessibility or Test Results, metric labels such as Score or Failed tests, status icons, and measured values such as `95.5%` are produced by Cypress Cloud from the assertion type and the run." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "A run's checks use the App Quality configuration saved at the time the run is recorded. When you " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/configuration/overview.md#Setting-Configuration", "children": [ { "type": "text", "value": "regenerate" } ] }, { "type": "text", "value": " a run, its checks are evaluated again against the configuration saved at that time, so you can adjust a check and see its effect on an existing run without recording again." } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Check statuses" } ] }, { "type": "table", "align": [ null, null ], "children": [ { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "Status" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Meaning" } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`passed`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Every assertion passed." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`failed`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "At least one assertion failed." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`errored`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "No assertion failed, but at least one is `errored` or `unavailable`, so the check couldn't confirm a pass." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`not-applicable`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The check's `include` and `exclude` rules don't select this run, so it wasn't evaluated." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "While a run is still processing, a check can also be `pending` (waiting on data) or `failing` (at least one assertion has already failed while others are still pending). A failed assertion never recovers, so a `failing` check always ends as `failed`." } ] }, { "type": "heading", "depth": 3, "children": [ { "type": "text", "value": "Assertion statuses" } ] }, { "type": "table", "align": [ null, null ], "children": [ { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "Status" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "Meaning" } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`passed`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The measured value met the threshold." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`failed`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The measured value missed the threshold." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`errored`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The data the assertion needs couldn't be read, for example because a report failed to load or a run-level score wasn't produced for the run." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`unavailable`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The run has no data for the assertion by design: the product it reads isn't enabled or processed for the run, or the project doesn't have access to the assertion type (such as `run.flakyTests.max` without flaky test detection)." } ] } ] }, { "type": "tableRow", "children": [ { "type": "tableCell", "children": [ { "type": "text", "value": "`pending`" } ] }, { "type": "tableCell", "children": [ { "type": "text", "value": "The assertion is still waiting for its run results or report to finish processing. Appears while evaluation is in progress; `getCheckResults()` waits until assertions leave this status." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Keep these behaviors in mind when designing checks:" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "A failure outranks an error. A check with one `failed` and one `errored` assertion is `failed`." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "An `unavailable` assertion makes its check `errored`, not `passed`. If a check mixes products and some runs don't produce one of them, scope the check with `include` and `exclude`, or split it into one check per product." } ] } ] } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "Using check results in CI" } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "To act on check results outside Cypress Cloud, such as blocking a merge or a deploy, fetch them in CI with `getCheckResults()` from the `@cypress/extract-cloud-results` module. Install it as described in the " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/connect-and-extend/results-api.md#Installation", "children": [ { "type": "text", "value": "Results API" } ] }, { "type": "text", "value": " guide, which also lists the " }, { "type": "link", "title": null, "url": "/llm/markdown/accessibility/connect-and-extend/results-api.md#Supported-CI-Providers", "children": [ { "type": "text", "value": "supported CI providers" } ] }, { "type": "text", "value": ". During Early Access, `getCheckResults()` is the supported way to access these results in CI, and matches how many teams already access UI Coverage or Cypress Accessibility results. Cypress Cloud CLI and MCP support are planned." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Run it in the same CI job that recorded the run, after `cypress run --record`:" } ] }, { "type": "list", "ordered": true, "start": 1, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "It identifies the run from the CI provider's environment variables, so you don't pass a run number." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "It waits until Cypress Cloud has finished evaluating the run's checks, including any Cypress Accessibility and UI Coverage reports they read. It checks every 30 seconds for up to 15 minutes." } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "It returns each check with its status and the status of each assertion, in the order your configuration lists them." } ] } ] } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "Results include each assertion's configured threshold and its status, but not the measured value. To show the score or count, link to the run's Properties tab in Cypress Cloud." } ] }, { "type": "code", "lang": null, "meta": null, "value": "const { getCheckResults } = require('@cypress/extract-cloud-results')\n\nconst results = await getCheckResults({\n projectId: 'abc123', // optional if the CYPRESS_PROJECT_ID env is set\n recordKey: '...', // optional if the CYPRESS_RECORD_KEY env is set\n runTags: ['release'], // optional; narrows the match when a CI job records more than one run\n})\n// {\n// runNumber: 57,\n// runUrl: 'https://cloud.cypress.io/projects/abc123/runs/57',\n// runStatus: 'failed',\n// checksStatus: 'evaluated', // or 'not-configured' when the project has no checks\n// checks: [\n// {\n// name: 'release-gate',\n// status: 'failed', // passed, failed, errored, or not-applicable\n// assertionResults: [\n// { type: 'accessibility.score.min', value: 98, status: 'failed' },\n// { type: 'run.failedTests.max', value: 0, status: 'passed' },\n// ],\n// },\n// ],\n// }" }, { "type": "paragraph", "children": [ { "type": "text", "value": "`getCheckResults()` throws if it can't find the run, if evaluation errors, or if evaluation doesn't finish in time. More status values may be added, so handle values you don't recognize." } ] }, { "type": "paragraph", "children": [ { "type": "text", "value": "To post each applicable check as a GitHub commit status and make it a required status check on the pull request, see " }, { "type": "link", "title": null, "url": "/llm/markdown/cloud/guides/report-run-checks-github.md", "children": [ { "type": "text", "value": "Report Run Checks as GitHub pull request statuses" } ] }, { "type": "text", "value": "." } ] }, { "type": "heading", "depth": 2, "children": [ { "type": "text", "value": "See also" } ] }, { "type": "list", "ordered": false, "start": null, "spread": false, "children": [ { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/cloud/guides/report-run-checks-github.md", "children": [ { "type": "text", "value": "Report Run Checks as GitHub pull request statuses" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/cloud/guides/quarantine-problematic-specs.md", "children": [ { "type": "text", "value": "Quarantine problematic specs with Run Checks" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/cloud/faq.md#Run-Checks", "children": [ { "type": "text", "value": "Cypress Cloud FAQ: Run Checks" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/accessibility/configuration/overview.md", "children": [ { "type": "text", "value": "Cypress Accessibility configuration" } ] }, { "type": "text", "value": " - every App Quality configuration option, including `views` that `viewMatch` matches against" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/ui-coverage/configuration/overview.md", "children": [ { "type": "text", "value": "UI Coverage configuration" } ] } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/cloud/features/flaky-test-management.md", "children": [ { "type": "text", "value": "Flaky Test Management" } ] }, { "type": "text", "value": " - the flake detection `run.flakyTests.max` reads" } ] } ] }, { "type": "listItem", "spread": false, "checked": null, "children": [ { "type": "paragraph", "children": [ { "type": "link", "title": null, "url": "/llm/markdown/app/references/command-line.md#cypress-run-tag-lt-tag-gt", "children": [ { "type": "text", "value": "`cypress run --tag`" } ] }, { "type": "text", "value": " - the run tags that check rules match" } ] } ] } ] } ] }, "token_estimate": 3753 }