DEV Community: Mike Shi

5 Ways to Better Debug Failing Cypress Tests in CI

Mike Shi — Thu, 19 Jan 2023 20:20:44 +0000

Debugging Cypress tests when they fail only in CI can be a challenging task. With the right tools and techniques, it’s possible to gather the debug info needed to fix a test, instead of just hitting retry one more time.

We’ll walk through a few tips you can use to pull more debug information out of your Cypress tests when they fail in CI. If you'd like, you can read the original version of this article directly on our blog.

Add Debug Data via `cy.log`

cy.log is a powerful tool to help annotate your tests with debug information for both local and remote development. cy.log output will show up in the command log UI and console.log of the test browser. This means in CI, information in cy.log can be seen in the screenshots or videos.

You can use cy.log to anotate what part of the test is running, or to print out test state like the ID of the fake user that's being used in the test to correlate with backend logs.

While this is the easiest method of adding additional debug information, the fact that it’s limited to being shown in videos or screenshots makes its utility
limited.

Output Debug Data via stdout

Often times debugging starts with looking through the logs of your CI output. However, you’ll quickly realize that regular console.log in Cypress tests won’t actually show up in your CI logs. While cy.log can help surface information inside of a video or screenshot, we can actually push data directly into standard output/CI logs.

We’ll need to do this via introducing a new cy.task to push debug information from your test script running in the browser, to the Node process that can output the information to the CLI.

In your Cypress config file (cypress.config.js) you’ll need to define the following :

const { defineConfig } = require('cypress');

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      on('task', {
        log: message => {
          console.log(message);
          return null;
        },
      });
    },
  },
});

Now within your test, you can easily call cy.task("log", ...anyDebugDataHere); to output debug information straight into your CI logs.

Install Terminal Report Plugin

Cypress terminal report is an open source Cypress plugin that outputs actions, intercepted requests, console messages and errors directly to stdout in a convenient format. It’s a more-complete version of using either cy.task or cy.log to output basic debug information to the CLI and can help surface more debug information for tricky test cases.

Getting it installed is really easy:

Run npm i --save-dev cypress-terminal-report
Install the plugin (in cypress.config.js)

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      require('cypress-terminal-report/src/installLogsPrinter')(on);
    },
  },
});

Install the support file (in cypress/support/e2e.js)

require('cypress-terminal-report/src/installLogsCollector')();

Afterwards you’ll be able to see action logs and other debug information in your stdout when a test fails.

Output Verbose Debug Logs from Cypress

If the test failure is particularly tricky, you can try enabling debug logs in Cypress to check for issues around excess memory/CPU usage (which can cause flakes) or specific network errors.

You can enable network logging by prepending DEBUG=cypress:net* in front of your Cypress run command like the following: DEBUG=cypress:net* yarn run cypress run --browser chrome

You can enable memory/CPU logging by prepending DEBUG=cypress:server:util:process_profiler in front of your Cypress run command.

However, these log messages are typically extremely verbose and can be difficult to sift through, so I’d only recommend trying to use these as a last resort, and keeping them disabled normally to keep your CI logs usable.

Install DeploySentinel Cypress Debugger Plugin

The DeploySentinel Cypress debugger plugin automatically collects every DOM snapshot, console logs, network request, memory usage, and more automatically and displays it within a cypress open-like debugging interface. It makes it easier to sift through and correlate mountains of debug information that can be put out by the tools above.

Getting it installed is as easy as any other Cypress plugin:

Run npm i --save-dev @deploysentinel/cypress-debugger
Install the plugin (in cypress.config.js)

const { defineConfig } = require('cypress');
module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      [on, config] = require('@deploysentinel/cypress-debugger/plugin')(
        on,
        config,
      );
    },
  },
});

Install the support file (in cypress/support/e2e.js)

import '@deploysentinel/cypress-debugger/support';

Run with a DeploySentinel API key (get started for free)

CYPRESS_DEPLOYSENTINEL_KEY=YOUR_API_KEY npx cypress run --browser chrome

From there, you’ll be able to debug test failures directly in the DeploySentinel web app, with all the telemetry you can get locally, and more!

Setting up Cypress with Github Actions and Heroku Review Apps

Mike Shi — Thu, 13 Oct 2022 19:03:29 +0000

I’ve spent this past weekend setting up Cypress tests for a project hosted on Heroku. I used Heroku's preview environments along with Github Actions to quickly set up Cypress to run against PRs in just a few minutes. Let's dive into what it takes to get it set up!

Set Up Heroku Review Apps

First we’ll want to enable review apps on Heroku, fortunately it’s super easy to get started. Heroku’s docs are available here.

The most important part is to choose the predictable review app URL
option - this makes it easy for your Cypress to know how to connect to your Heroku preview instance. We'll use this later in our Github workflow.

Set up Github Action Workflow

With our Github action workflow, we’re going to use the
Github Action context to access the current PR number, which will let us specify the review app URL. So we can specify the review app URL like:
https://cy-heroku-review-pr-${{github.event.pull_request.number }}.herokuapp.com

In this example, I chose cy-heroku-review as my review app prefix, you'll need to replace it with the predictable URL pattern you chose earlier in this guide.

Additionally, we’ll be using the wait-on npm package to have our
Github runner wait until the URL is available on Heroku before triggering Cypress. This allows us to have more control on waiting for the review app to be available than just letting Cypress retry itself (ex. if you want a longer wait period).

on:
  pull_request:
    branches: [main]
jobs:
  test-heroku:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - run: yarn install
      # Here we're waiting for the Heroku preview instance to boot up
      - name: Wait for Server
        run:
          npx wait-on https://cy-heroku-review-pr-${{
          github.event.pull_request.number }}.herokuapp.com
      # Here we're running the test, specifying the baseUrl via the CLI
      - name: Run E2E Tests
        run:
          yarn test --config baseUrl=https://cy-heroku-review-pr-${{
          github.event.pull_request.number }}.herokuapp.com
        # Adding the DeploySentinel debugger
        # So we can easily debug test failures in case anything goes wrong 😉
        env:
          CYPRESS_DEPLOYSENTINEL_KEY: ${{ secrets.CYPRESS_DEPLOYSENTINEL_KEY }}
          ELECTRON_EXTRA_LAUNCH_ARGS: '--remote-debugging-port=40500'

Run the Test!

With the review app configured, and the Github action workflow in place, you can open a new PR and see the new workflow kick off and run your tests against a Heroku preview instance!

I have a full demo repo available here to check out.

Feel free to message me any thoughts/questions @ mike@deploysentinel.com

No Tears Guide to Creating E2E Test Scripts for Playwright & Puppeteer

Mike Shi — Thu, 17 Mar 2022 20:29:30 +0000

Originally published on the DeploySentinel Blog

Getting browser automation scripts to do exactly what you want can feel like a never-ending battle between you and your code. I still remember vividly when I was up late one night years ago writing a Selenium script for a change I was making. I was banging my head against the table as I was running out of Selenium incantations to cast on my test to make it work.

Fast forward to today, I’m writing up this guide on a few tricks I wish I knew at the time, to quickly craft reliable E2E tests, instead of crying over error logs and wrangling clueless automation scripts.

Use a Recorder

A recorder is a tool that can record the browser actions you take, and turn those actions into a fully-working automation script. So you can start coding out your test, by simply stepping through your user flow as a user would.

I know there are people who might be skeptical, but hear me out. Just as automation frameworks have dramatically improved over time, test recorders have gotten some serious love over time as well.

I find test recorders pretty useful for a few reasons:

No more hunting for selectors - There’s no reason to be hunting for selectors yourself, when you can automate the work away. The best recorders use selector logic that will give you stable selectors, rather than trying to enumerate every div and class name on the way to your element. (We’ll also talk more about selectors later!)
No more manually writing commands - Just like selectors, there’s no need to manually type out every “click” “fill” and “waitFor” command, when you can get all of that for free by just simply walking through your web pages.
Best practices are built in - While recorders won’t generate perfect code, they will have best-practices built-in. Instead of needing to worry about using “fill” instead of “type”, or remembering how to await for text in Puppeteer, you can just let the recorder generate code that’s aligned with best practices, so you can focus on the rest of your work.

Ultimately, using a test recorder can give you a head start, save you from doing the boring parts of testing, and let you focus on the harder parts of perfecting a test automation script.

Here’s a few recorders that can help you get started:

DeploySentinel Recorder: A Chrome/Firefox Extension that generates Playwright/Puppeteer scripts (Disclaimer: I am one of the authors, but only because I wanted to build the most accurate recorder available)
Playwright Codegen: A CLI tool included in Playwright.
Puppeteer Chrome DevTool Recorder: A recorder built into Chrome for Puppeteer.
Headless Recorder: A Chrome Extension that can automate some simpler actions.

Use Stable Selectors

As dynamic CSS classes and elements are the norm today, it can be difficult to try to pick selectors that are robust to a rapidly changing code base, or even a rapidly changing web page due to async logic.

Especially if your project uses something like Tailwinds, styled-components, or even a JS framework like React or Vue that use async logic to render elements, you might be struggling to figure out how to target the right element reliably.

First Solution: Test IDs

The best way to combat these issues is to explicitly introduce a stable selector to elements you’re looking to test, and nothing beats stability like introducing test IDs to your application.

The concept is very simple: for elements you need to interact with, append a data-test-id=”my-element” attribute to the HTML element. In your automation scripts, you can easily target the element with

await page.click('[data-test-id="my-element"]')

Now you’ll never need to worry about your selectors breaking the next time your team decides to change button colors or re-builds the application with a new minified class name.

However, this requires adding new attributes to elements you need to target. Depending on how open the application owners are to adding these new attributes, it may be difficult to rely on them. If test IDs aren’t possible, I’d fall back to the next best solution.

Second Solution: Accessibility Selectors

Luckily with accessibility becoming a higher priority for web applications, more and more critical elements that need to be interacted with might already have a machine-friendly label attached to it.

Usually you’ll see attributes such as aria-label, alt, or title for elements you want to interact with. Those attributes tend to be more stable than CSS classes and can serve as a good stop-gap measure until you’re able to implement test IDs for the elements you need to test.

A script that utilizes these attributes might look like

await page.click('[alt="Main Logo"]')

Last Solution: Text Content Selectors

If you’re testing an application that might not have all the accessibility selectors built out yet, and haven’t had time to implement test IDs, the last solution you can look towards is targeting elements by text content.

At first glance, it might sound like an incredibly fragile proposition. Indeed it can be for certain elements, but for others it may be the best stable solution available. Can you remember the last time your team updated the “Sign In” button text on your application?

For elements with non-dynamic text content, usually buttons or input placeholders, text content can be a fairly reliable way of target elements.

Luckily in Playwright, it’s incredibly easy to target elements by text like so:

page.click('text=Sign In')

In Puppeteer, you’ll need to dip into XPaths to target elements by text:

await page.waitForXPath('//*[contains(., "Sign In")]')
const [element] = await page.$x('//*[contains(., "Sign In")]');
await element.click();

Automate Selector Picking

These best practices of stable selectors mentioned here and more are already built-in to DeploySentinel Recorder’s selector picking logic. So you don’t have to hunt for a specific test ID or accessibility selector.

Flip On the Debug Features

If you’re banging your head against a test script trying to figure out why it isn’t working, whipping out the debugging modes is probably the fastest way to find out why your script isn’t doing what you want.

Playwright’s Inspector and Trace Viewer

With Playwright, it’s incredibly easy to append PWDEBUG=1 in front of your script to pull up Playwright Inspector, where it’ll be able to step through everything Playwright is doing in great detail during the test. If there’s a step you have issues with, you can add await page.pause() to pause the test run so you can inspect the page at that point in time.

If you’re executing the script in a remote environment, you can take advantage of Playwright’s Trace Viewer which records detailed logs and DOM snapshots before and after every action.

If you’re using DeploySentinel to run your test - Playwright traces are captured by default and viewable at any time to debug test runs.

Headed Mode and Slow Mo

In general you can also enable headed mode with slow motion enabled to visually see what your script is doing. Both Playwright and Puppeteer support this with just two extra lines of code. See the docs for Playwright and Puppeteer here.

The DeploySentinel Recorder will always have these two options commented out but inserted as part of every script generated to make it easy to debug locally.

Enable Chrome Dev Tools

Lastly if there’s an issue that requires you to look at network requests or browser logs, you can have Playwright and Puppeteer open the Chrome dev tools panel on browser launch so all logs and network requests are captured from the start automatically for you. See the Playwright docs here or the dev tools section of Puppeteer’s debugging docs.

You’re All Set!

I hope this list of tips helps you out in creating testing scripts for Puppeteer or Playwright.

If you’re looking to run tests easily and affordably - you can try DeploySentinel for free, with many of the above best practices included for you with no extra effort.