See our founder and CTO, Brian Mann, speak at All Things Open on October 14th in Raleigh, NC

Learn More

Visual Testing

Improve this doc

What you’ll learn

  • How visual testing complements functional testing
  • How to implement visual diffing yourself or using 3rd party service
  • How to ensure the application is in consistent state before capturing an image

Functional vs visual testing

Cypress is a functional Test Runner. It drives the web application the way a user would, and checks if the app functions as expected: if the expected message appears, an element is removed, or a CSS class is added after the appropriate user action. A typical Cypress test, for example, can check if a toggled “Todo” item gets a class of “completed” after the .toggle is checked:

it('completes todo', () => {
  // opens TodoMVC running at "baseUrl"
  cy.visit('/')
  cy.get('.new-todo').type('write tests{enter}')
  cy.contains('.todo-list li', 'write tests')
    .find('.toggle').check()
  cy.contains('.todo-list li', 'write tests')
    .should('have.class', 'completed')
})
Passing Cypress functional test

Cypress does NOT see how the page actually looks though. For example, Cypress will not see if the CSS class completed grays out the label element and adds a strike-through line.

Completed item style

You could technically write a functional test asserting the CSS properties using the have.css assertion, but these may quickly become cumbersome to write and maintain, especially when visual styles rely on a lot of CSS styles.

cy.get('.completed').should('have.css', 'text-decoration', 'line-through')
cy.get('.completed').should('have.css', 'color', 'rgb(217,217,217)')

Your visual styles may also rely on more than just CSS, perhaps you want to ensure an SVG or image has rendered correctly or shapes were correctly drawn to a canvas.

Luckily, Cypress gives a stable platform for writing plugins that can perform visual testing.

Typically such plugins take an image snapshot of the entire application under test or a specific element, and then compare the image to a previously approved baseline image. If the images are the same (within a set pixel tolerance), it is determined that the web application looks the same to the user. If there are differences, then there has been some change to the DOM layout, fonts, colors or other visual properties that needs to be investigated.

For example, one can use the cypress-plugin-snapshots plugin and catch the following visual regression:

.todo-list li.completed label {
  color: #d9d9d9;
  /* removed the line-through */
}
it('completes todo', () => {
  cy.visit('/')
  cy.get('.new-todo').type('write tests{enter}')
  cy.contains('.todo-list li', 'write tests')
    .find('.toggle')
    .check()
  cy.contains('.todo-list li', 'write tests')
    .should('have.class', 'completed')
  // run 'npm i cypress-plugin-snapshots -S'
  // capture the element screenshot and
  // compare to the baseline image
  cy.get('.todoapp').toMatchImageSnapshot({
    threshold: 0.001,
  })
})

This open source plugin compares the baseline and the current images side by side within the Cypress Test Runner if pixel difference is above the threshold; notice how the baseline image (Expected result) has the label text with the line through, while the new image (Actual result) does not have it.

Baseline vs current image

Like most image comparison tools, the plugin also shows a difference view on mouse hover:

Highlighted changes

Tooling

There are several published, open source plugins, listed in the Visual Testing plugins section, and several commercial companies have developed visual testing solutions on top of the Cypress Test Runner listed below.

Open source

Listed in the Visual Testing plugins section.

Applitools

https://applitools.com
Resource Description
Official docs Applitools’ Cypress documentation
Tutorial Applitools’ Cypress tutorial
Webinar Creating a Flawless User Experience, End-to-End, Functional to Visual – Practical Hands-on Session, a webinar recorded together with Cypress and Applitools
Blog Testing a chart with Cypress and Applitools

Percy

https://percy.io
Resource Description
Official docs Percy’s Cypress documentation
Tutorial Percy’s Cypress tutorial
Webinar Cypress + Percy = End-to-end functional and visual testing for the web, a webinar recorded together with Cypress and Percy.io
Blog The companion blog for the Cypress + Percy webinar
Slides The companion slides for the Cypress + Percy webinar
Blog Testing how an application renders a drawing with Cypress and Percy

Do It Yourself

Even if you decide to skip using a 3rd party image storage and comparison service, you can still perform visual testing. Follow the example Visual Regression testing with Cypress and cypress-image-snapshot tutorial and do it all yourself.

You will want to consider the development costs of implementing a visual testing tool yourself versus using an external 3rd party provider. Storing, reviewing and analyzing image differences are non-trivial tasks and they can quickly become a chore when going with a DIY solution.

Best practices

As a general rule there are some best practices when visual testing.

Recognize the need for visual testing

assertions that verify style properties

cy.get('.completed').should('have.css', 'text-decoration', 'line-through')
  .and('have.css', 'color', 'rgb(217,217,217)')
cy.get('.user-info').should('have.css', 'display', 'none')
...

If your end-to-end tests become full of assertions checking visibility, color and other style properties, it might be time to start using visual diffing to verify the page appearance.

DOM state

Best Practice: Take a snapshot after you confirm the page is done changing.

For example, if the snapshot command is cy.mySnapshotCommand:

Incorrect Usage

// the web application takes time to add the new item,
// sometimes it takes the snapshot BEFORE the new item appears
cy.get('.new-todo').type('write tests{enter}')
cy.mySnapshotCommand()

Correct Usage

// use a functional assertion to ensure
// the web application has re-rendered the page
cy.get('.new-todo').type('write tests{enter}')
cy.contains('.todo-list li', 'write tests')
// great, the new item is displayed,
// now we can take the snapshot
cy.mySnapshotCommand()

Timestamps

Best Practice: Control the timestamp inside the application under test.

Below we freeze the operating system’s time to Jan 1, 2018 using cy.clock() to ensure all images displaying dates and times match.

const now = new Date(2018, 1, 1)

cy.clock(now)
// ... test
cy.mySnapshotCommand()

Application state

Best Practice: Use cy.fixture() and network mocking to set the application state.

Below we stub network calls using cy.route() to return the same response data for each XHR request. This ensures that the data displayed in our application images does not change.

cy.server()
cy.route('/api/items', 'fixture:items').as('getItems')
// ... action
cy.wait('@getUsers')
cy.mySnapshotCommand()

Visual diff elements

Best Practice: Use visual diffing to check individual DOM elements rather than the entire page.

Targeting specific DOM element will help avoid visual changes from component “X” breaking tests in other unrelated components.

Component testing

Best Practice: Use Component Testing plugins to test the individual components functionality in addition to end-to-end and visual tests.

See also