AWS CodeBuild

With AWS CodeBuild Amazon Web Services (AWS) offers developers "...a fully managed build service that compiles your source code, runs unit tests, and produces artifacts that are ready to deploy" allowing teams to "pay only for the build time you use".

Detailed documentation is available in the AWS CodeBuild Documentation.

Basic Setup

The example below is basic CI setup and job using the AWS CodeBuild to run Cypress tests within the Electron browser. This AWS CodeBuild configuration is placed within buildspec.yml.

## buildspec.yml
version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: latest
    commands:
      - npm ci
  pre_build:
    commands:
      - npm run cy:verify
      - npm run cy:info
  build:
    commands:
      - npm run start:ci &
      - npx cypress run --record

How this buildspec works:

  • On push to this repository, this job will provision and start AWS-hosted Amazon Linux instance with Node.js for running the outlined pre_build and build for the declared commands within the commands section of the configuration.
  • AWS CodeBuild will checkout our code from our GitHub repository.
  • Finally, our buildspec.yml configuration will:
    • Install npm dependencies
    • Start the project web server (npm start:ci)
    • Run the Cypress tests within our GitHub repository within Electron.

Testing in Chrome and Firefox with Cypress Docker Images

AWS CodeBuild offers a build-list strategy of different job configurations for a single job definition.

The build-list strategy offers a way to specify an image hosted on DockerHub or the Amazon Elastic Container Registry (ECR).

The Cypress team maintains the official Docker Images for running Cypress locally and in CI, which are built with Google Chrome and Firefox. For example, this allows us to run the tests in Firefox by passing the --browser firefox attribute to cypress run.

## buildspec.yml
version: 0.2

## AWS CodeBuild Batch configuration
## https://docs.aws.amazon.com/codebuild/latest/userguide/batch-build-buildspec.html
## Define build to run using the "cypress/browsers:node12.14.1-chrome85-ff81" image from the Cypress Amazon ECR Public Gallery
batch:
  fast-fail: false
  build-list:
    - identifier: cypress-e2e-tests
      env:
        variables:
          IMAGE: public.ecr.aws/cypress-io/cypress/browsers:node12.14.1-chrome85-ff81

phases:
  install:
    runtime-versions:
      nodejs: latest
    commands:
      - npm ci
  pre_build:
    commands:
      - npm run cy:verify
      - npm run cy:info
  build:
    commands:
      - npm run start:ci &
      - npx cypress run --record --browser firefox

Caching Dependencies and Build Artifacts

Caching with AWS CodeBuild directly can be challenging.

The Build caching in AWS CodeBuild document offers details on local or Amazon S3 caching.

Per the documentation, "Local caching stores a cache locally on a build host that is available to that build host only". This will not be useful during parallel test runs.

The "Amazon S3 caching stores the cache in an Amazon S3 bucket that is available across multiple build hosts". While this may sound useful, in practice the upload of cached dependencies can take some time. Furthermore, each worker will attempt to save it's dependency cache to Amazon S3, which increases build time significantly.

Beyond the scope of this guide, but AWS CodePipeline may be of use to cache the initial source, dependencies and build output for use in AWS CodeBuild jobs using AWS CodePipeline Input and Output Artifacts.

Reference the AWS CodePipeline integration with CodeBuild and multiple input sources and output artifacts sample example for details on how to configure a CodePipeline with an output artifact.

Parallelization

The Cypress Dashboard offers the ability to parallelize and group test runs along with additional insights and analytics for Cypress tests.

AWS CodeBuild offers a build-matrix strategy for declaring different job configurations for a single job definition. The build-matrix strategy provides an option to specify a container image for the job. Jobs declared within a build-matrix strategy can run in parallel which enables us run multiples instances of Cypress at same time as we will see later in this section.

The Cypress team maintains the official Docker Images for running Cypress locally and in CI, which are built with Google Chrome and Firefox. This allows us to run the tests in Firefox by passing the --browser firefox attribute to cypress run.

Parallelizing the build

To setup multiple containers to run in parallel, the build-matrix configuration uses a set of variables (CY_GROUP_SPEC and WORKERS) with a list of items specific to each group for the build.

The fields are delimited by a pipe (|) character as follows:

## Group Name | Browser | Specs | Cypress Configuration options (optional)

'UI - Chrome - Mobile|chrome|cypress/tests/ui/*|viewportWidth=375,viewportHeight=667'

The build-matrix will run all permutations delimited items.

batch:
  fast-fail: false
  build-matrix:
    # ...
    dynamic:
      env:
        # ...
        variables:
          CY_GROUP_SPEC:
            - 'UI - Chrome|chrome|cypress/tests/ui/*'
            - 'UI - Chrome - Mobile|chrome|cypress/tests/ui/*|viewportWidth=375,viewportHeight=667'
            - 'API|chrome|cypress/tests/api/*'
            - 'UI - Firefox|firefox|cypress/tests/ui/*'
            - 'UI - Firefox - Mobile|firefox|cypress/tests/ui/*|viewportWidth=375,viewportHeight=667'

During the install phase, we utilize shell scripting with the cut command to assign values from the delimited CY_GROUP_SPEC passed to the worker into shell variables that will be used in the build phase when running cypress run.

batch:
  # ...

phases:
  install:
    commands:
      - CY_GROUP=$(echo $CY_GROUP_SPEC | cut -d'|' -f1)
      - CY_BROWSER=$(echo $CY_GROUP_SPEC | cut -d'|' -f2)
      - CY_SPEC=$(echo $CY_GROUP_SPEC | cut -d'|' -f3)
      - CY_CONFIG=$(echo $CY_GROUP_SPEC | cut -d'|' -f4)
      - npm ci
## ...

To parallelize the runs, we need to add an additional variable to the build-matrix strategy, WORKERS.

batch:
  fast-fail: false
  build-matrix:
    # ...
    dynamic:
      env:
        # ...
        variables:
          CY_GROUP_SPEC:
            # ...
          WORKERS:
            - 1
            - 2
            - 3
            - 4
            - 5

Finally, the script variables are passed to the call to cypress run.

phases:
  install:
    # ...
  build:
    commands:
      - npm start:ci &
      - npx cypress run --record --parallel --browser $CY_BROWSER --ci-build-id $CODEBUILD_INITIATOR --group "$CY_GROUP" --spec "$CY_SPEC" --config "$CY_CONFIG"

Using the Cypress Dashboard with AWS CodeBuild

In the AWS CodeBuild configuration we have defined in the previous section, we are leveraging three useful features of the Cypress Dashboard:

  1. Recording test results with the --record flag to the Cypress Dashboard:

  2. Parallelizing test runs and optimizing their execution via intelligent load-balancing of test specs across CI machines with the --parallel flag.

  3. Organizing and consolidating multiple cypress run calls by labeled groups into a single report within the. Cypress Dashboard. In the example above we use the --group "UI - Chrome" flag (for the first group) to organize all UI tests for the Chrome browser into a group labeled "UI - Chrome" inside the Cypress Dashboard report.

Cypress Real World Example with AWS CodeBuild

A complete CI workflow against multiple browsers, viewports and operating systems is available in the Real World App (RWA).

Clone the Real World App (RWA) and refer to the buildspec.yml file.