UI CoveragePremium Solution

Block pull requests and set policies

Cypress UI Coverage reports are generated server-side in Cypress Cloud, based on test artifacts uploaded during execution. This ensures there is no performance impact on your Cypress test runs, but also means that nothing in your Cypress pipeline will fail due to coverage issues that are detected. Failing a build is a fully opt-in step based on your handling of the results in your CI process.

Using the Results API

The Cypress UI Coverage Results API allows you to access UI Coverage results post-test run, enabling workflows like blocking pull requests or triggering alerts based on specific coverage criteria. This involves adding a dedicated UI Coverage verification step to your CI pipeline. With a Cypress helper function, you can automatically fetch the report for the relevant test run within the CI build context.

Implementing a status check

The Results API offers full flexibility to analyze results and take tailored actions. It can also integrate with status checks on pull requests, allowing you to block merges when coverage thresholds are not met.

Defining policies in the verification step

The Results API Documentation provides detailed guidance on the API's capabilities. Here's a simplified example demonstrating how to enforce a minimum coverage threshold:

const { getUICoverageResults } = require('@cypress/extract-cloud-results')

// Fetch UI Coverage results
getUICoverageResults().then((results) => {
  const { summary, views } = results

  // Verify overall coverage
  if (summary.coverage < 80) {
    throw new Error(
      `Project coverage is ${summary.coverage}%, below the minimum threshold of 80%.`
    )
  }

  // Verify critical view coverage
  const criticalViews = [/login/, /checkout/]
  views.forEach((view) => {
    if (
      criticalViews.some((pattern) => pattern.test(view.displayName)) &&
      view.coverage < 95
    ) {
      throw new Error(
        `Critical view "${view.displayName}" coverage is ${view.coverage}%, below the required 95%.`
      )
    }
  })
})

By examining the results and customizing your response, you gain maximum control over how to handle coverage gaps. Leverage CI environment context, such as tags, to fine-tune responses to specific coverage outcomes.

Using Profiles for PR-specific configuration

You can use Profiles to apply different configuration settings for pull request runs versus regression runs. This allows you to:

• Use a narrow, focused configuration for PR runs that blocks merges based on critical coverage thresholds
• Maintain a broader configuration for regression runs that tracks all coverage gaps for long-term monitoring
• Apply team-specific configurations when multiple teams share the same Cypress Cloud project

For example, you might configure a profile named aq-config-pr that excludes non-critical pages and focuses only on the most important coverage areas, while your base configuration includes all pages for comprehensive regression tracking.

cypress run --record --tag "aq-config-pr"

This approach ensures that PR checks are fast and focused, while still maintaining comprehensive reporting for your full test suite.

Comparing against a baseline

Comparing current results against a stored baseline allows you to detect only new untested elements that have been introduced, while ignoring existing coverage gaps. This approach helps you focus on regressions in test coverage and track improvements over time.

This is particularly useful in CI/CD pipelines where you want to fail builds only when new untested elements are introduced, allowing you to address existing coverage gaps incrementally without blocking deployments.

Baseline structure

In our example the baseline is a JSON object that captures the state of untested elements from a specific run. It includes:

• runNumber: The run number used as the baseline reference
• views: An object mapping view display names to arrays of view identifiers and their untested elements counts
• runUrl: The link to the cloud run that generated this report

{
  "runNumber": 68086,
  "runUrl": "https://cloud.cypress.io/projects/ypt4pf/runs/68086",
  "views": {
    "/": { testedElements: 55 },
    "/login": { testedElements: 3 },
    "/products/*": { testedElements: 3 },
  }
}

Why use untested element counts instead of UI Coverage percentage scores?

The UI Coverage score is expressed as based on a ratio of what was and was not tested over time, within what was rendered in Cypress during the run. Since testing one new element might reveal many more elements that aren't tested yet, the score isn't useful for a fine-grained baseline comparison between runs. Comparing the number of tested elements gives a more accurate sense of whether one run has added or removed coverage when compared to another, and is a better predictor of what information would be in the report.

Complete example

The following example demonstrates how to compare current results against a baseline, detect new views with untested elements, identify views where coverage has improved, and generate a new baseline on every run so that it's easy to copy and update if needed.

scripts/compareUICoverageBaseline.js

require('dotenv').config()

const { getUICoverageResults } = require('@cypress/extract-cloud-results')
const fs = require('fs')

// Define your baseline - this should be stored and updated as your application improves
// Running the script once with no baseline will generate a baseline for you
const baseline = {
  runNumber: undefined,
  runUrl: undefined,
  views: [],
}

// Function to compare coverage between current results and baseline
const compareTestedElementsWithBaseline = (currentResults, baselineData) => {
  const testedElementsIssues = []
  const testedElementsImprovements = []
  const newPages = []

  // Check if current results has views
  if (!currentResults.views || !Array.isArray(currentResults.views)) {
    console.log(
      'Warning: Current results do not contain a valid "views" array. Skipping comparison.'
    )
    return {
      testedElementsIssues,
      testedElementsImprovements,
      newPages,
      hasChanges: false,
    }
  }

  const currentViews = currentResults.views

  // Check if baseline data has the expected structure
  if (!baselineData.views) {
    console.log(
      'Warning: Baseline data does not contain "views" property. Skipping comparison.'
    )
    console.log('Current baseline structure:', Object.keys(baselineData))
    return {
      testedElementsIssues,
      testedElementsImprovements,
      newPages,
      hasChanges: true,
    }
  }

  const baselineViews = baselineData.views

  // Ensure baselineViews is an array
  if (!Array.isArray(baselineViews)) {
    console.log('Warning: Baseline views is not an array. Skipping comparison.')
    return {
      testedElementsIssues,
      testedElementsImprovements,
      newPages,
      hasChanges: true,
    }
  }

  // Create a map of baseline testedElementsCount by displayName for quick lookup
  const baselineTestedElementsMap = {}
  baselineViews.forEach((view) => {
    baselineTestedElementsMap[view.displayName] = view.testedElementsCount
  })

  // Create a map of current views by displayName to check for missing pages
  const currentViewsMap = {}
  currentViews.forEach((view) => {
    currentViewsMap[view.displayName] = view.testedElementsCount
  })

  // Compare each current view with baseline
  currentViews.forEach((currentView) => {
    const pageName = currentView.displayName
    const currentTestedElements = currentView.testedElementsCount
    const pageExistsInBaseline = pageName in baselineTestedElementsMap

    if (pageExistsInBaseline) {
      const baselineTestedElements = baselineTestedElementsMap[pageName]
      if (currentTestedElements < baselineTestedElements) {
        testedElementsIssues.push({
          page: pageName,
          currentTestedElements: currentTestedElements,
          baselineTestedElements: baselineTestedElements,
          difference: baselineTestedElements - currentTestedElements,
        })
      } else if (currentTestedElements > baselineTestedElements) {
        testedElementsImprovements.push({
          page: pageName,
          currentTestedElements: currentTestedElements,
          baselineTestedElements: baselineTestedElements,
          difference: currentTestedElements - baselineTestedElements,
        })
      }
    } else {
      // New page not in baseline
      newPages.push({
        page: pageName,
        testedElementsCount: currentTestedElements,
      })
    }
  })

  // Check for pages in baseline that are missing from current results
  // These are treated as regressions (0 tested elements)
  baselineViews.forEach((baselineView) => {
    const pageName = baselineView.displayName
    if (!(pageName in currentViewsMap)) {
      testedElementsIssues.push({
        page: pageName,
        currentTestedElements: 0,
        baselineTestedElements: baselineView.testedElementsCount,
        difference: baselineView.testedElementsCount,
        isMissing: true,
      })
    }
  })

  const hasChanges =
    testedElementsIssues.length > 0 ||
    testedElementsImprovements.length > 0 ||
    newPages.length > 0

  return {
    testedElementsIssues,
    testedElementsImprovements,
    newPages,
    hasChanges,
  }
}

function generateUICoverageBaseline(results) {
  try {
    // Generate simplified baseline with only runNumber, runUrl, and views with displayName and testedElementsCount
    return {
      runNumber: results.runNumber,
      runUrl: results.runUrl,
      views: results.views.map((view) => ({
        displayName: view.displayName,
        testedElementsCount: view.testedElementsCount,
      })),
    }
  } catch (error) {
    console.error('Error generating UI Coverage baseline:', error)
    return null
  }
}

getUICoverageResults()
  .then((results) => {
    // Compare tested elements count with baseline
    const {
      testedElementsIssues,
      testedElementsImprovements,
      newPages,
      hasChanges,
    } = compareTestedElementsWithBaseline(results, baseline)

    if (hasChanges) {
      // Generate and log the new baseline values if there has been a change
      const newBaseline = generateUICoverageBaseline(results)
      console.log('\nTo use this run as the new baseline, copy these values:')
      console.log(JSON.stringify(newBaseline, null, 2))
      fs.writeFileSync(
        'new-UICbaseline.json',
        JSON.stringify(newBaseline, null, 2)
      )
    }

    // Log tested elements regressions
    if (testedElementsIssues.length > 0) {
      console.error('\n❌ TESTED ELEMENTS REGRESSION DETECTED!')
      console.error(
        'The following pages have fewer tested elements than the baseline:'
      )
      console.error('')

      testedElementsIssues.forEach((issue) => {
        console.error(`  📄 ${issue.page}`)
        if (issue.isMissing) {
          console.error(
            `     ⚠️  MISSING FROM REPORT (treated as 0 tested elements)`
          )
          console.error(
            `     Current: 0 | Baseline: ${issue.baselineTestedElements} | Difference: -${issue.difference}`
          )
        } else {
          console.error(
            `     Current: ${issue.currentTestedElements} | Baseline: ${issue.baselineTestedElements} | Difference: -${issue.difference}`
          )
        }
        console.error('')
      })

      console.error(
        `Total pages with tested elements regression: ${testedElementsIssues.length}`
      )
    }

    // Log tested elements improvements
    if (testedElementsImprovements.length > 0) {
      console.log('\n✅ TESTED ELEMENTS IMPROVEMENTS DETECTED!')
      console.log(
        'The following pages have more tested elements than the baseline:'
      )
      console.log('')

      testedElementsImprovements.forEach((improvement) => {
        console.log(`  📄 ${improvement.page}`)
        console.log(
          `     Current: ${improvement.currentTestedElements} | Baseline: ${improvement.baselineTestedElements} | Difference: +${improvement.difference}`
        )
        console.log('')
      })

      console.log(
        `Total pages with tested elements improvement: ${testedElementsImprovements.length}`
      )
    }

    // Log new pages
    if (newPages.length > 0) {
      console.log('\n📝 NEW PAGES DETECTED (not in baseline):')
      newPages.forEach((page) => {
        console.log(
          `  📄 ${page.page} - Tested Elements: ${page.testedElementsCount}`
        )
      })
      console.log(`Total new pages: ${newPages.length}`)
    }

    // Summary
    if (!hasChanges) {
      console.log(
        '\n✅ All pages meet or exceed baseline tested elements count!'
      )
    } else if (testedElementsIssues.length === 0) {
      console.log('\n✅ No tested elements regressions detected!')
    }

    // Exit with error code if there are regressions
    if (testedElementsIssues.length > 0) {
      process.exit(1)
    }
  })
  .catch((error) => {
    console.error('Error getting UI coverage results:', error)
    process.exit(1)
  })

Key concepts

New untested elements

A new untested element situation occurs when a view has untested elements in the current run but did not have untested elements in the baseline. This represents a regression in test coverage that needs to be addressed. The script will fail the build if any views with new untested elements are detected.

Resolved untested elements

A resolved untested element situation occurs when a view had untested elements in the baseline but no longer has untested elements in the current run. This represents an improvement in test coverage. The script reports these but does not fail the build, allowing you to track progress.

View-level comparison

Untested elements are tracked per view (URL pattern or component), allowing you to see exactly which pages or components have coverage regressions or improvements. This granular tracking makes it easier to identify where new tests are needed or where coverage has improved.

Best practices

When to update the baseline

Update your baseline when:

• You've added tests to cover previously untested elements and want to prevent regressions
• You've accepted certain coverage gaps as known issues that won't block deployments
• You want to track coverage improvements over time

Store the baseline in version control so it's versioned alongside your code and accessible in CI environments.

Handling partial reports

If a run is cancelled or incomplete, the Results API may return a partial report. Consider checking summary.isPartialReport before comparing against the baseline, as partial reports may not include all views and could produce false positives.

Managing baseline across branches

You may want different baselines for different branches (e.g., main vs feature branches). Consider storing baselines in branch-specific files or using environment variables to specify which baseline to use.

Storing the baseline

Common approaches for storing baselines:

• Version control: Commit the baseline JSON file to your repository
• CI artifacts: Store baselines as build artifacts that can be retrieved in subsequent runs
• External storage: Use cloud storage or a database for baselines if you need more sophisticated versioning

info

This baseline comparison approach complements the Branch Review UI feature, which provides visual comparisons between runs. The programmatic approach is ideal for CI/CD automation, while Branch Review is better suited for manual investigation and code review workflows.