Cypress CloudFree Trial

Parallelization

info

What you'll learn

• How Cypress balances spec files across multiple machines
• How to run tests in parallel across multiple machines
• How to group test runs and why you might want to
• How to complete a run early with the Run Completion API
• How to visualize parallelization and groups in Cypress Cloud

If your project has a large number of tests, it can take a long time for tests to complete running serially on one machine. Running tests in parallel across many virtual machines can save your team time and money when running tests in Continuous Integration (CI).

Cypress can run recorded tests in parallel across multiple machines. While parallel tests can also technically run on a single machine, we do not recommend it since this machine would require significant resources to run your tests efficiently.

This guide assumes you already have your project running and recording within Continuous Integration. If you have not set up your project yet, check out our Continuous Integration guide. If you are running or planning to run tests across multiple browsers (Firefox, Chrome, or Edge), we also recommend checking out our Cross Browser Testing guide for helpful CI strategies when using parallelization.

Splitting up your test suite

Cypress' parallelization strategy is file-based, so in order to utilize parallelization, your tests will need to be split across separate files.

Cypress will assign each spec file to an available machine based on our balance strategy. Due to this balance strategy, the run order of the spec files is not guaranteed when parallelized.

Turning on parallelization

1. Refer to your CI provider's documentation on how to set up multiple machines to run in your CI environment.

2. Once multiple machines are available within your CI environment, you can pass the --parallel key to cypress run to have your recorded tests parallelized.

cypress run --record --key=abc123 --parallel

info

Running tests in parallel requires the --record flag be passed. This ensures Cypress can properly collect the data needed to parallelize future runs. This also gives you the full benefit of seeing the results of your parallelized tests in Cypress Cloud. If you have not set up your project to record, check out our setup guide.

CI parallelization interactions

During parallelization mode, Cypress Cloud interacts with your CI machines to orchestrate the parallelization of a test run via load-balancing of specs across available CI machines by the following process:

1. CI machines contact Cypress Cloud to indicate which spec files to run in the project.
2. A machine opts in to receiving a spec file to run by contacting Cypress.
3. Upon receiving requests from a CI machine, Cypress calculates the estimated duration to test each spec file.
4. Based on these estimations, Cypress distributes (load-balances) spec files one-by-one to each available machine in a way that minimizes overall test run time.
5. As each CI machine finishes running its assigned spec file, more spec files are distributed to it. This process repeats until all spec files are complete.
6. Upon completion of all spec files, Cypress waits for a configurable amount of time before considering the test run as fully complete. This is done to better support grouping of runs.

In short: each Cypress instance sends a list of the spec files to Cypress Cloud, which sends back one spec at a time to each application to run.

Parallelization process

Example

The examples below are from a run of our Kitchen Sink Example project. You can see the results of this run on Cypress Cloud.

Without parallelization

In this example, a single machine runs all spec files serially. Below is an example .circleci/config.yml using the Cypress CircleCI Orb:

.circleci/config.yml

version: 2.1
orbs:
  cypress: cypress-io/cypress@6
workflows:
  build:
    jobs:
      - cypress/run:
          start-command: 'npm run start'
          cypress-command: 'npx cypress run --record'

Cypress runs all 19 spec files one by one alphabetically. It takes 1:51 to complete all of the tests.

1x-electron, Machine #1
--------------------------
-- actions.cy.js (14s)
-- aliasing.cy.js (1s)
-- assertions.cy.js (1s)
-- connectors.cy.js (2s)
-- cookies.cy.js (2s)
-- cypress_api.cy.js (3s)
-- files.cy.js (2s)
-- local_storage.cy.js (1s)
-- location.cy.js (1s)
-- misc.cy.js (4s)
-- navigation.cy.js (3s)
-- network_requests.cy.js (3s)
-- querying.cy.js (1s)
-- spies_stubs_clocks.cy.js (1s)
-- traversal.cy.js (4s)
-- utilities.cy.js (3s)
-- viewport.cy.js (3s)
-- waiting.cy.js (5s)
-- window.cy.js (1s)

info

Notice that when adding up the spec's run times (0:55), they add up to less than the total time for the run to complete (1:51) . There is extra time in the run for each spec: starting the browser, encoding and uploading the video to the dashboard, requesting the next spec to run.

With parallelization

When we run the same tests with parallelization, Cypress uses its load balance strategy to order to specs to run based on the spec's previous run history. During the same CI run as above, we ran all tests again, but this time with parallelization across 2 machines. By adding parallelism: 2 and the --parallel flag to the .circleci/config.yml:

.circleci/config.yml

version: 2.1
orbs:
  cypress: cypress-io/cypress@6
workflows:
  build:
    jobs:
      - cypress/run:
          parallelism: 2
          start-command: 'npm run start'
          cypress-command: 'npx cypress run --record --parallel'

It finished in 59 seconds.

2x-electron, Machine #1, 9 specs          2x-electron, Machine #2, 10 specs
--------------------------------          -----------------------------------
-- actions.cy.js (14s)                    -- waiting.cy.js (6s)
-- traversal.cy.js (4s)                   -- navigation.cy.js (3s)
-- misc.cy.js (4s)                        -- utilities.cy.js (3s)
-- cypress_api.cy.js (4s)                 -- viewport.cy.js (4s)
-- cookies.cy.js (3s)                     -- network_requests.cy.js (3s)
-- files.cy.js (3s)                       -- connectors.cy.js (2s)
-- location.cy.js (2s)                    -- assertions.cy.js (1s)
-- querying.cy.js (2s)                    -- aliasing.cy.js (1s)
-- location.cy.js (1s)                    -- spies_stubs_clocks.cy.js (1s)
                                          -- window.cy.js (1s)

The difference in running times and machines used is very clear when looking at the Machines View on Cypress Cloud. Notice how the run parallelized across 2 machines automatically ran all specs based on their duration, while the run without parallelization did not.

Parallelizing our tests across 2 machines saved us almost 50% of the total run time, and we can further decrease the build time by adding more machines.

Grouping test runs

Multiple cypress run calls can be labeled and associated to a single run by passing in the --group <name> flag, where name is an arbitrary reference label. The group name must be unique within the associated test run.

info

For multiple runs to be grouped into a single run, it is required for CI machines to share a common CI build ID environment variable. Typically these CI machines will run in parallel or within the same build workflow or pipeline, but it is not required to use Cypress parallelization to group runs. Grouping of runs can be utilized independently of Cypress parallelization.

info

Cross Browser Testing

Grouping test runs with or without parallelization is a useful mechanism when implementing a CI strategy for cross browser testing. Check out the Cross Browser Testing guide to learn more.

Grouping by browser

You can test your application against different browsers and view the results under a single run within Cypress Cloud. Below, we name our groups the same name as the browser being tested:

The first group can be called Windows/Chrome 69.

cypress run --record --group Windows/Chrome-69 --browser chrome

The second group can be called Mac/Chrome 70.

cypress run --record --group Mac/Chrome-70 --browser chrome

The third group can be called Linux/Electron. Electron is the default browser used in Cypress runs.

cypress run --record --group Linux/Electron

Grouping to label parallelization

We also have the power of Cypress parallelization with our groups. For the sake of demonstration, let's run a group to test against Chrome with 2 machines, a group to test against Electron with 4 machines, and another group to test against Electron again, but only with one machine:

cypress run --record --group 1x-electron

cypress run --record --group 2x-chrome --browser chrome --parallel

cypress run --record --group 4x-electron --parallel

The 1x, 2x, 4x group prefix used here is an adopted convention to indicate the level of parallelism for each run, and is not required or essential.

info

The number of machines dedicated for each cypress run call is based on your CI configuration for the project.

Labeling these groups in this manner helps up later when we review our test runs in Cypress Cloud, as shown below:

Grouping by spec context

Let's say you have an application that has a customer facing portal, guest facing portal and an administration facing portal. You could organize and test these three parts of your application within the same run:

One group can be called package/admin:

cypress run --record --group package/admin --spec 'cypress/e2e/packages/admin/**/*'

Another can be called package/customer:

cypress run --record --group package/customer --spec 'cypress/e2e/packages/customer/**/*'

The last group can be called package/guest:

cypress run --record --group package/guest --spec 'cypress/e2e/packages/guest/**/*'

This pattern is especially useful for projects in a monorepo. Each segment of the monorepo can be assigned its own group, and larger segments can be parallelized to speed up their testing.

Linking CI machines for parallelization or grouping

A CI build ID is used to associate multiple CI machines to one test run. This identifier is based on environment variables that are unique to each CI build, and vary based on CI provider. Cypress has out-of-the-box support for most of the commonly-used CI providers, so you would typically not need to directly set the CI build ID via the --ci-build-id flag.

CI Build ID environment variables by provider

Cypress currently uses the following CI environment variables to determine a CI build ID for a test run:

Provider	Environment Variable
AppVeyor	APPVEYOR_BUILD_NUMBER
AWS CodeBuild	CODEBUILD_INITIATOR
Azure Pipelines	BUILD_BUILDNUMBER
Bamboo	bamboo_buildNumber
Bitbucket	BITBUCKET_BUILD_NUMBER
Buildkite	BUILDKITE_BUILD_ID
Circle	CIRCLE_WORKFLOW_ID, CIRCLE_BUILD_NUM
Codeship	CI_BUILD_NUMBER
Codeship Basic	CI_BUILD_NUMBER
Codeship Pro	CI_BUILD_ID
Drone	DRONE_BUILD_NUMBER
GitLab	CI_PIPELINE_ID
Jenkins	BUILD_NUMBER
Semaphore	SEMAPHORE_EXECUTABLE_UUID
Travis	TRAVIS_BUILD_ID

You can pass a different value to link agents to the same run. For example, if you are using Jenkins and think the environment variable BUILD_TAG is more unique than the environment variable BUILD_NUMBER, pass the BUILD_TAG value via CLI --ci-build-id flag.

cypress run --record --parallel --ci-build-id $BUILD_TAG

Run Completion Delay

During parallelization mode or when grouping runs, Cypress will wait for a specified amount of time before completing the test run in case any more relevant work remains. This is to compensate for various scenarios where CI machines could be backed-up in a queue.

This waiting period is called the Run Completion Delay and it begins after the last known CI machine has completed as shown in the diagram below:

This delay is 60 seconds by default, but is configurable in the Cypress Cloud project settings page.

Run Completion API

caution

This API is experimental. Breaking changes are unlikely but may be made if necessary.

The Run Completion Delay is a safety buffer. For teams with non-deterministic build timing (for example, when running Cypress tests using Nx), that buffer often has to be long so that pauses between test groups do not cause the run to complete early.

Without keeping the run open to wait for new groups, a late group attempting to join an already-completed Cypress Cloud run would cause an error.

A long Run Completion Delay has only one drawback: once all your expected tests have been recorded, the delay becomes a needless wait for Cypress Cloud to resolve the run's "completed" status. This can slow down notifications and delay the finalization of UI Coverage and Cypress Accessibility reports.

That's where the Run Completion API comes in. It lets you signal to Cypress Cloud, in real time during a run, that it does not need to wait for more groups. The run will complete when all of the currently-active run groups have finished, or immediately if nothing is in progress.

The end result: no tradeoff needed between covering potential recording gaps in your CI setup and having Cypress Cloud resolve your runs as quickly as possible with correct grouping.

info

Using this API sends an imperative signal, not a force. The API never cuts off a run mid-flight. If groups are still in progress when you call it, the request is deferred and the run completes (with no delay) the instant it stabilizes. See Outcomes below.

Endpoint

POST https://api.cypress.io/runs/complete

Requests are authenticated with your project ID and record key, the same credentials used to record the run.

Request body

The request body is JSON. projectId and recordKey are always required, and the run is identified by exactly one of runUrl or runNumber.

Property	Type	Required	Description
projectId	string	Yes	The project ID of the run.
recordKey	string	Yes	A valid record key for the project.
runUrl	string	Conditional	The Cypress Cloud run URL (Cypress prints this when a run starts). Provide this or runNumber.
runNumber	number	Conditional	The run's sequential number within the project. Provide this or runUrl.
expectedGroupCount	number	No	The number of groups the run is expected to have before it completes. When set, the run only completes once that many groups have finished.

Example: fetch

const response = await fetch('https://api.cypress.io/runs/complete', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    projectId: 'abc123',
    recordKey: 'f858a2bc-b469-4e48-be67-0876339ee7e1',
    runUrl: 'https://cloud.cypress.io/projects/abc123/runs/42',
    // runNumber: 42, // optional, in lieu of runUrl
    // expectedGroupCount: 4, // optional
  }),
})

const result = await response.json()
console.log(result) // e.g. { outcome: 'completed' }

Example: curl

curl -X POST https://api.cypress.io/runs/complete \
  -H 'Content-Type: application/json' \
  -d '{
    "projectId": "abc123",
    "recordKey": "f858a2bc-b469-4e48-be67-0876339ee7e1",
    "runUrl": "https://cloud.cypress.io/projects/abc123/runs/42"
  }'

Finding the run URL or run number

Both identifiers come from the results Cypress produces when a run is recorded. You don't need to construct them yourself.

• From the Module API: the object that cypress.run() resolves with includes a runUrl field (present whenever the run was recorded).

  const cypress = require('cypress')
  
  const results = await cypress.run({ record: true, key: 'your-record-key' })
  
  // "https://cloud.cypress.io/projects/abc123/runs/42"
  const runUrl = results.runUrl

• From the after:run Node event: the results argument is the same object resolved by the Module API, so results.runUrl is available there as well. See the example below for a full after:run handler that calls this API.

Pass that value as runUrl. If you'd rather send runNumber, it's the trailing segment of the run URL: .../runs/42 corresponds to runNumber: 42.

info

runUrl is only present when the run was recorded (--record). Runs that aren't recorded aren't tracked by Cypress Cloud and can't be completed through this API.

Where to call the API in a parallelized run

caution

after:run fires once per machine, so every machine sees the same runUrl and would each call the endpoint. Read this section to ensure consistent, expected outcomes in CI.

Calling from inside after:run is fine when you pass a known expectedGroupCount with every call. Because the API defers until that many groups have finished (and absorbs any still joining), it doesn't matter which machine finishes first or how often the API is called from the run. The run only completes once all expected groups are in. Use this pattern when your group count is fixed and available to the handler (for example, injected as an environment variable by CI).

Firing a bare "complete now" request from all of them is different: whichever machine finishes its own specs first would ask to complete a run whose other groups may not have joined yet. For that reason, if you do not pass expectedGroupCount, the most reliable place to make the call is a dedicated post-run step in your CI pipeline that runs after all parallel jobs have finished: one machine, one call, once you truly know the run is done.

Example: calling from after:run with expectedGroupCount

tip

To try this API without changing your CI pipeline, you can hard-code expectedGroupCount in the handler instead of reading it from the environment.

The example below completes the run from inside after:run. It's safe to fire from every machine only because expectedGroupCount is supplied from an environment variable that CI sets to the run's fixed number of groups.

cypress.config.js

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  // setupNodeEvents can be defined in either
  // the e2e or component configuration
  e2e: {
    setupNodeEvents(on, config) {
      on('after:run', async (results) => {
        // `results` is undefined in `cypress open` mode, and `runUrl` is only
        // set when the run was recorded. Bail out otherwise.
        if (!results?.runUrl) {
          return
        }

        // The total number of `--group` names in this run, provided by CI.
        // Passing this is what makes the call safe to fire from every machine.
        const expectedGroupCount = Number(process.env.EXPECTED_GROUP_COUNT)

        if (!expectedGroupCount) {
          return
        }

        await fetch('https://api.cypress.io/runs/complete', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({
            projectId: results.config.projectId,
            recordKey: process.env.CYPRESS_RECORD_KEY,
            runUrl: results.runUrl,
            expectedGroupCount,
          }),
        })
      })
    },
  },
})

cypress.config.ts

import { defineConfig } from 'cypress'

export default defineConfig({
  // setupNodeEvents can be defined in either
  // the e2e or component configuration
  e2e: {
    setupNodeEvents(on, config) {
      on('after:run', async (results) => {
        // `results` is undefined in `cypress open` mode, and `runUrl` is only
        // set when the run was recorded. Bail out otherwise.
        if (!results?.runUrl) {
          return
        }

        // The total number of `--group` names in this run, provided by CI.
        // Passing this is what makes the call safe to fire from every machine.
        const expectedGroupCount = Number(process.env.EXPECTED_GROUP_COUNT)

        if (!expectedGroupCount) {
          return
        }

        await fetch('https://api.cypress.io/runs/complete', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({
            projectId: results.config.projectId,
            recordKey: process.env.CYPRESS_RECORD_KEY,
            runUrl: results.runUrl,
            expectedGroupCount,
          }),
        })
      })
    },
  },
})

Outcomes

The endpoint always responds with a JSON body containing an outcome field. The HTTP status reflects whether the run reached a terminal state as a result of the request.

Status	outcome	Meaning
200	completed	The run was ready and has been completed immediately, skipping the completion delay.
200	already-complete	The run had already completed. The request is idempotent: no change was made.
202	deferred	The run is not ready yet (groups are still in progress). The request was recorded; the run will complete with no delay once it stabilizes.
202	run-not-active	There is no live recording session for the run. It will finalize on its own.

When a deferred response is returned for a request that included an expectedGroupCount, the body also echoes the progress toward that target:

{
  "outcome": "deferred",
  "completedGroupCount": 2,
  "expectedGroupCount": 4
}

Here, completedGroupCount is the number of groups that have actually finished (not merely registered).

Visualizing parallelization and groups in Cypress Cloud

You can see the result of each spec file that ran within Cypress Cloud in the run's Specs tab. Specs are visualized within a Timeline, Bar Chart, and Machines view.

Timeline View

The Timeline View charts your spec files as they ran relative to each other. This is especially helpful when you want to visualize how your tests ran chronologically across all available machines.

Bar Chart View

The Bar Chart View visualizes the duration of your spec files relative to each other.

Machines View

The Machines View charts spec files by the machines that executed them. This view enables you to evaluate the contribution of each machine to the overall test run.

Next Steps

• Cypress Real World App runs parallelized CI jobs across multiple operating systems, browsers, and viewport sizes.
• Continuous Integration Guide
• Cross Browser Testing Guide
• Blog: Run Your End-to-end Tests 10 Times Faster with Automatic Test Parallelization
• CI Configurations in Kitchen Sink Example