DEV Community: Checkly

The Advent of Monitoring, Day 3: Easy Monitoring for Self-Hosted Projects with Checkly

Sara Miteva — Thu, 14 Dec 2023 13:42:33 +0000

This is the third part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

This article was written by Daniel Paulus, Checkly's Director of Engineering.

When it comes to running self-hosted services or side projects, monitoring is key. But, who has the time to set up a complex monitoring system?

We want to deliver cool software and not be busy with configuring Prometheus servers or Grafana Dashboards. That’s where synthetic monitoring fits in perfectly – it's straightforward to set up and reliably keeps an eye on things.

Setting Up Multi-Step API Checks

I love to use Checkly for this because of the great Developer Experience it gives me. It's particularly handy with its latest feature: multi-step API checks (still in beta). Multi step checks allow you to write code and use HTTP requests to implement an arbitrary number of requests, organized in steps, and custom logic to monitor any HTTP based service. This is ideal for more advanced monitoring needs without getting too complicated.

The task at hand was to set up basic monitoring for my self-hosted ClickHouse database. For those who want to jump straight in, all the necessary code is available on GitHub. Just clone it, set up ch_pass (clickhouse password), ch_user (clickhouse user name) and ch_url (’https://IP:port’) account variables in your checkly account and run npx checkly deploy

If you prefer a step-by-step guide, here’s how I did it:

We will use Clickhouse’ HTTP interface to run queries against it to check the database health.

First, create a Checkly project. It's as easy as running npm create checkly and following the instructions. Then, you'll need four files in your checks directory:

clickhouse.check.ts:

import { MultiStepCheck, Frequency } from "checkly/constructs";
import path from "node:path";

//this defines a check. The actual code is in the spec files
new MultiStepCheck("clickhouse-check-1", {
  name: "Clickhouse Version Check",
  runtimeId: "2023.09",
  frequency: Frequency.EVERY_10M,
  locations: ["us-east-1", "eu-west-1"],
  code: {
    entrypoint: path.join(__dirname, "clickhouse.spec.ts"),
  },
});

new MultiStepCheck("clickhouse-check-2", {
  name: "Clickhouse Free Diskspace",
  runtimeId: "2023.09",
  frequency: Frequency.EVERY_12H,
  locations: ["us-east-1", "eu-west-1"],
  code: {
    entrypoint: path.join(__dirname, "clickhouse-disk.spec.ts"),
  },
});

clickhouse.spec.ts contains a very basic set of checks. First it pings the Clickhouse health endpoint to check if the server is running at all. If that succeeds, we run the most basic query, a SELECT version() and make sure the version string is returned as expected. Now we know that Clickhouse is responsive and ready to go.

import { test, expect } from "@playwright/test";
import { baseUrl, queryClickhouse } from "./utils";

const clickHouseVersion = "23.9.1.1854";

async function pingClickhouse(request) {
  const response = await request.get(`${baseUrl}/ping`);
  expect(response.ok()).toBeTruthy();
  const msg = await response.text();
  expect(msg.trim()).toBe("Ok.");
}

test("check clickhouse", async ({ request }) => {
  await test.step("ping clickhouse", async () => {
    await pingClickhouse(request);
  });

  await test.step("check clickhouse version", async () => {
    const response = await queryClickhouse("SELECT version()", request);
    expect(response[0]).toBe(clickHouseVersion);
  });
});

clickhouse-disk.spec.ts is a very simple check to make sure Clickhouse has at least 5GB of free disk space left. If not, the check will fail and we will get an alert from Checkly. Clickhouse offers a lot of interesting metrics through it’s query interface so you can monitor many other things. Take a look: https://clickhouse.com/blog/clickhouse-debugging-issues-with-system-tables

import { test, expect } from "@playwright/test";
import { queryClickhouse } from "./utils";

const diskSpaceQuery = `
SELECT 
    name, 
    free_space / 1024 / 1024 / 1024 AS free_space_gb
FROM system.disks;

`;

test("check clickhouse", async ({ request }) => {
  await test.step("check clickhouse diskspace", async () => {
    const response = await queryClickhouse(diskSpaceQuery, request);
    const row = response[0].split("\t");
    console.log(`Disk ${row[0]} has ${row[1]}GB free`);
    //make sure we got more than 5GB left
    expect(parseFloat(row[1])).toBeGreaterThan(5);
  });
});

utils.js contains a very simple Clickhouse response parser.. I know there is a lot of room for improvement here. But you know, I want to build apps not monitoring ;-) As you can see, you need to set up the environment variables for user name, password and the Clickhouse http url in your Checkly env variables for this to work.

import { expect } from "@playwright/test";
export const baseUrl = process.env.ch_url;

export async function queryClickhouse(query, request) {
  const buff = Buffer.from(query, "utf-8");
  const response = await request.post(`${baseUrl}`, {
    headers: {
      "X-ClickHouse-User": process.env.ch_user,
      "X-ClickHouse-Key": process.env.ch_pass,
    },
    data: buff,
  });

  const buf = await response.body();
  expect(response.ok()).toBeTruthy();
  const res = new String(buf).toString();
  const resArray = res.split("\n");
  resArray.pop();
  return resArray;
}

Once you have your files ready, run npx checkly login to login to your account and then give it a try by running npx checkly test. If that works and you Clickhouse cluster responds, it is time to unite testing with monitoring and deploy your checks to Checkly for continuous active monitoring. Deploy your checks using npx checkly deploy.

And we’re done! This setup will continuously monitor your ClickHouse database, ensuring it’s up and running all the time. How cool is it that you can put the monitoring code next to your application code and deploy changes that easily?

In case your service does encounter an issue, it's important to get notified quickly. Checkly offers various alerting methods, including SMS, phone calls, Slack, and Pagerduty. You can set these up by checking out Checkly's Alert Channels documentation.

With Checkly, setting up monitoring for your self-hosted projects is a breeze, giving you both peace of mind and more time to focus on other aspects of your projects.

This is the third part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

This article was written by Daniel Paulus, Checkly's Director of Engineering.

The Advent of Monitoring, Day 2: Debugging Dashboard Outages with Checkly's API Checks

Sara Miteva — Tue, 12 Dec 2023 17:39:38 +0000

This is the second part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

This post was written by Daniel Paulus, Checkly's Director of Engineering.

We encountered a tricky issue with our public dashboards: they were experiencing sporadic outages, happening about once every two days. The infrequency and unpredictability of these outages made them particularly challenging to diagnose.

Our Hypothesis

Initially, we tried to correlate the outages with our logs, looking at the failure times reported by our browser check failures. Unfortunately, this method didn't yield any useful insights.

This is when we tried to use high-frequency API checks, running every 10 seconds. Our existing setup involved a browser check that ran hourly, which was sufficient under normal circumstances but not detailed enough to catch these intermittent issues.

ChatGPT to the Rescue

The issue with our public dashboards is, of course, already fixed for a long time, so to simulate a similar scenario, I used ChatGPT4 to generate a simple nodeJS server for me using the following prompt:

Write a simple nodejs based rest endpoint that always returns a body "OK" and a 200 status code without using a framework. The endpoint should return right away. There should be a loop that runs every 10s and with a 0.1% chance, will cause the endpoint to return HTTP 500 for a time window of 30-60 seconds. Afterwards it will go back to returning 200.

Now all I had to do was to save the resulting code to a file and then run it with NodeJS. To publicly expose it for testing purposes a simple

ngrok http 3000

was the perfect solution.

Configuring API Checks with Checkly

Next, I configured two API checks with Checkly, one running every 5 minutes and one running every 10 seconds.

The screenshots really illustrate the benefits of temporarily or permanently using higher frequency checks. The 5 min frequency check did not detect all the error windows, and also does not help with understanding how long the service was down. Things might seem much better here compared to when you start probing the API every 10s.

5 minutes:

10 seconds:

The results are quite revealing. The high-frequency API check shows almost exactly when the issue started, when it stopped and how long it took. It identifies all the 30-60s random errors. In real-world scenarios, this makes it much easier to find the correct logs and more importantly, also understand how big the problem actually is.

By leveraging Checkly's API checks for high-resolution failure timing, we were able to identify and subsequently address a problem that our standard monitoring approach had missed. This experience underscored the importance of synthetic monitoring with higher frequency checks, especially for issues that occur sporadically. With Checkly's help, we were able to root cause and fix the dashboard's sporadic outages and could enhance the reliability of our service.

This is the second part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

This post was written by Daniel Paulus, Checkly's Director of Engineering.

The Advent of Monitoring, Day 1: What Are Synthetics and Why They Are Needed

Sara Miteva — Mon, 11 Dec 2023 15:55:31 +0000

This is the first part of Checkly's 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

This post was written by Daniel Paulus, Checkly's Director of Engineering.

Hey there! Here is my take on what synthetic monitoring means and why it’s awesome!

I think it’s a very complicated word for a very straightforward concept. In fact, I am convinced, that once you've used it, you will never want to live without it.

Let’s start first with e2e testing: In essence, what you do is, define a series of steps interacting with your web app or REST API to simulate real user behavior.

For web apps, that means your test clicks buttons and types text while making assertions along the way.

For REST APIs, you check that the actual HTTP requests are fast enough and contain the expected responses. This way, you make sure core user flows are working as expected when you test your code before deploying it to production.

Any refactoring that might break a ton of unit tests will not affect e2e tests unless the core user flow is actually broken. So, they give you the ultimate signal if your system behaves as it should. If you use a modern e2e testing framework like Playwright, your tests will also be quite fast and not flaky, unlike Selenium or other old testing tools.

How does all of this relate to synthetics?

Well, the general idea is to promote those e2e tests you have created, to continuously run against your production services! Now you have the best signal there is to understand if users can actually use your service, all the time, in your production environment. You will know when things break before your users inform you. Consider also the following questions you are finally getting answers to:

The unit tests all worked, but does the feature actually work on production?
Is the auth provider having an outage and people cannot log in, although your own software is correct?
Is any other third-party dependency that is slow or broken?
It works in Germany, but does it work in California? Is there some regional blocking in place?

Take a look at the screenshot below, for example. You can immediately spot which part of the system is affected. Compare that to a list of error logs or multiple metrics dashboards and you will understand what I mean when I say synthetics give you a clear signal compared to all the noise.

Ultimately, synthetic monitoring means monitoring through simulated users, or testing in production, and is like your app's guardian angel. It's always there, quietly keeping watch, ensuring everything runs smoothly for your users.

For those in the software development scene, it's an invaluable ally, keeping your app top-notch and your users happy. With a modern tool like Checkly, leveraging Monitoring as Code, you can repurpose your existing e2e tests to run against production by writing them twice.

This is the first part of Checkly's 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

This post was written by Daniel Paulus, Checkly's Director of Engineering.

Using Playwright to Monitor Third-Party Resources That Could Impact User Experience

Stefan Judis — Wed, 15 Feb 2023 16:03:57 +0000

Today’s web consists of lots of 3rd party resources. Let it be your fonts, transformed and optimized media assets, or analytics and ad scripts, many sites out there include resources that they don’t own. Your website probably has a lot of those dependencies, too!

And while implementing third-party resources has downsides for performance and you should self-host your assets when possible, sometimes relying on external files is unavoidable. But be warned — using them can sometimes cause real headaches.

As Chris Coyier pointed out, external resources are often integrated too tightly into your own code, or implemented the wrong way, so that things can really go sideways. Ad blockers could block your external scripts or a service vendor could be down. Both situations can result in frontend downtime, poor UX, or entirely broken functionality on your end.

And while Chris asks if you could detect the cases of broken sites due to ad blockers with end-to-end testing, I think it’s more about all the 3rd party code running on the internet.

But heck yeah, testing and monitoring of failing external resources is possible!

Let’s have a look at an example that could be affected by 3rd party code and ways to monitor your stack with Playwright to guarantee that no external providers or ad blockers can mess with your business. 🫡

Examples of problematic external resource usage

But first, let’s look at patterns that can break your sites.

In my experience, there are two challenging 3rd party resource scenarios.

Slow and render-blocking resources

Years ago, I worked in an ecommerce startup and we were serving ad banners via an external provider. It was a quick solution to receive click statistics while providing marketers with a comfortable way to handle images on the site.

The banners were implemented with synchronous script elements above the fold.

<script type="text/javascript" src="http://some-provider.js"></script>

And we had plenty of these elements embedded in a slideshow. The scripts would load and replace containers with content coming from elsewhere. It worked beautifully… until it didn’t.

A synchronous script element stops the HTML parser and forces it to wait for the script to be downloaded and executed. The browser only continues parsing and rendering once this is done so that this script resulting in an embedded image can literally block anything on your site.

And that’s what exactly happened to me back in the day.

The script was loaded fairly quickly from Berlin, Germany and we didn’t notice any delays in rendering. But one day our third-party vendor experienced an outage. And interestingly, requests wouldn’t fail but just hang and time out eventually.

The result: a browser started rendering our site, stopped at the slideshow and waited for the scripts to time out. Our customers were looking at a white page until this happened and our site was entirely broken. A third-party vendor brought our frontend down because we implemented synchronous scripts. Be aware that every external dependency is another risk for your sites!

That’s why it’s generally better to self-host your assets and avoid critical render-blocking elements. No matter if you’re using ad tech, error monitoring scripts, or external stylesheets, make sure to double-check what happens when these resources fail. I doubt you want your system to be affected because of someone else's downtime.

That’s not the only situation where third-party resources can lead to trouble, though. Let’s come back to Chris’ example.

Tightly coupled JavaScript dependencies

Chris mentioned that he experiences broken sites on a regular basis. Some sites would fail in essential situations like making an order. The culprit: a thrown JavaScript exception.

That’s interesting because it’s unlikely that web developers aren’t testing their most important functionality. So what’s happening?

If you’re an ad blocker user there’s a high chance that this tiny browser extension is the reason. And that’s not the ad blocker’s fault. It’s doing what it’s supposed to do — blocking trackers (most likely third-party scripts). But the issue is often that developers expect these scripts to be loaded. For example, if your website’s JavaScript expects a sendTracking method to be available in the global window object, an ad blocker can easily mess with the site and cause JavaScript to fall on its nose.

Here’s Chris’ example:

// https://tracking-website.com/tracking-script.js
window.trackThing = function() {
  // report data
}

// /script/index.js
function init() {
  // this throws an exception if `tracking-script.js` was blocked
  trackThing("page loaded or something");
}
init();

If tracking-script.js fails or is blocked by a browser extension your code throws an exception and could blow up your entire app. If you’re implementing external JavaScript functionality, you should make sure that things are still functional in error cases because I bet you prefer processing an order over tracking it.

But how can you go around these two problems and guarantee that you won’t be running into frontend downtimes based on synchronous resources or “death by ad blocker” scenarios in the future?

How to test and monitor risky third-party resources with Playwright

Let’s take a simple HTML example running on my localhost:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Frontend downtime</title>
  <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0-alpha1/dist/css/bootstrap.min.css" rel="stylesheet"
integrity="sha384-GLhlTQ8iRABdZLl6O3oVMWSktQOp6b7In1Zl3/Jr59b6EGGoI1aFkw7cmDA6j6gD" crossorigin="anonymous">
</head>
<body>
  <h1>Hello world</h1>
  <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0-alpha1/dist/js/bootstrap.bundle.min.js"
integrity="sha384-w76AqPfDkMBDXo30jS1Sgez6pr3x5MlQ1ZAGC+nuZB+EYdgRZgiwxhTBTkF7CXvN"
crossorigin="anonymous"></script>
  <script>bootstrap.Alert.getOrCreateInstance('#myAlert')</script>
</body>
</html>

It includes Bootstrap’s CSS and JavaScript, renders the headline “Hello world” and tries to access bootstrap.Alert which is available in the global window scope.

Note that in this case, both external resources are render-blocking. The stylesheet has to be loaded before anything appears on the screen and even though the script is placed at the end of the document, it will still block rendering (there’s just nothing to render after it).

A quick Playwright test for this page could look like this:

// @ts-check
const { test, expect } = require("@playwright/test")

test("has the correct title", async ({ page }) => {
  await page.goto("http://localhost:8080")

  // evaluate the largest contentful paint
  const largestContentfulPaint = await page.evaluate(() => {
    return new Promise((resolve) => {
      new PerformanceObserver((l) => {
        const entries = l.getEntries()
        // the last entry is the largest contentful paint
        const largestPaintEntry = entries.at(-1)
        resolve(largestPaintEntry.startTime)
      }).observe({
        type: "largest-contentful-paint",
        buffered: true,
      })
    })
  })

  console.log(parseFloat(largestContentfulPaint))

  await expect(page).toHaveTitle(/Frontend downtime/)
})

Playwright navigates to localhost and evaluates the popular largest contentful paint metric by injecting custom JavaScript into the page. When you run the script with npx playwright test you’ll see the following in your terminal:

Running 1 test using 1 worker
[chromium] › example.spec.js:4:1 › has the correct title
CLP: 116.39999999850988ms

1 passed (837ms)

Wonderful! The largest contentful paint is a hundred milliseconds long and everything works. But what if the Bootstrap resources on jsdeliver.net are slow to respond?

How to emulate slow third-party resources with Playwright

Let’s find out and delay requests by ten seconds that aren’t fetching resources from the same origin with Playwright’s page.route method.

// @ts-check
const { test, expect } = require("@playwright/test")

test("works with slows resources", async ({ page }) => {
  page.route(
    "**",
    (route) =>
      new Promise((resolve) => {
        const requestURL = route.request().url()

        if (requestURL.match(/http:\/\/localhost:8080/)) {
          resolve(route.continue())
        } else {
          setTimeout(() => {
            console.log(`Delaying ${requestURL}`)
            resolve(route.continue())
          }, 10000)
        }
      })
  )
  await page.goto("http://localhost:8080")

  const largestContentfulPaint = await page.evaluate(() => {
    return new Promise((resolve) => {
      new PerformanceObserver((l) => {
        const entries = l.getEntries()
        // the last entry is the largest contentful paint
        const largestPaintEntry = entries.at(-1)
        resolve(largestPaintEntry.startTime)
      }).observe({
        type: "largest-contentful-paint",
        buffered: true,
      })
    })
  })

  console.log(`CLP: ${parseFloat(largestContentfulPaint)}ms`)

  // Expect a title "to contain" a substring.
  await expect(page).toHaveTitle(/Frontend downtime/)
})

The test still passes but its test duration skyrocketed from 800ms to 10 seconds and similarly, the largest contentful paint also joined at the ten seconds mark.

Running 1 test using 1 worker
[chromium] › example.spec.js:4:1 › has the correct title
Delaying https://cdn.jsdelivr.net/npm/bootstrap@5.3.0-alpha1/dist/css/bootstrap.min.css
Delaying https://cdn.jsdelivr.net/npm/bootstrap@5.3.0-alpha1/dist/js/bootstrap.bundle.min.js

CLP: 10186.10000000149ms

1 passed (11.0s)

What is going on? Let’s create a Playwright trace and debug this test.

There are two things to note in this Playwright trace: first, the test duration increased to ten seconds because Playwright’s page.goto waits for the page onload event which is heavily delayed by the slowed-down stylesheet.

And second, and probably worse than a slow test case, the user experience suffered tremendously because the page only started rendering after ten seconds.

After running a quick Playwright test, you now know that there are 3rd party resources that could really harm your overall performance and you can check if they’re necessary or if you should implement them in another way.

But what if these requests fail or are blocked by browser extensions?

How to block requests with Playwright

So far we’ve only been delaying third-party resources, but what happens when we block them entirely? Luckily, blocking resources is quickly done with page.route, too.

test("works with blocked 3rd party resources", async ({ page }) => {
  page.route("**", (route) => {
    const requestURL = route.request().url()

    if (requestURL.match(/http:\/\/localhost:8080/)) {
      route.continue()
    } else {
      route.abort()
    }
  })

  // monitor emitted page errors
  const errors = []
  page.on("pageerror", (error) => {
    errors.push(error)
  })

  await page.goto("http://localhost:8080")

  console.log(errors)
  expect(errors.length).toBe(0)
})

If you run the script the test case fails because the page emits pageerrors. The inline JavaScript tried to call a Bootstrap method that wasn’t available because of resource blocking similar to how an ad blocker would do it.

Running 1 tests using 1 workers
[chromium] › example.spec.js:43:1 › works with blocked 3rd party resources
[
  Error [ReferenceError]: bootstrap is not defined at http://localhost:8080/:18:11
]
1) [chromium] › example.spec.js:43:1 › works with blocked 3rd party resources ====================

Error: expect(received).toBe(expected) // Object.is equality
    Expected: 0
    Received: 1

    60 |
    61 |   console.log(errors)
  > 62 |   expect(errors.length).toBe(0)
       |                         ^
    63 | })
    64 |

Tracking page errors is a handy way to monitor your JavaScript in a synthetic lab setup. In the case of ad blockers and canceled requests, you could still perform your end-to-end tests and evaluate if a blocked script affects your site at all.

Either way, page.route is an easy-to-use way to emulate network conditions with Playwright.

So what’s next?

Here’s the takeaway: whenever you implement and rely on resources you don’t control, make sure that they’re not affecting or blowing up your application! People use browser extensions, networks are flaky, services go down… many things that can go wrong.

Know how your app handles slow responses or requests that might become victims of ad blockers.

Note: This post’s code snippets implement page request handling on a test-case basis for simplicity. I recommend looking into Playwright’s fixtures to not clutter your tests and focus on your end-to-end logic. Work with a nicely abstracted pageWithSlowThirdParties object instead!

But how should you go about these two scenarios then? Should you run all your deployment end-to-end tests with changed third-party network settings?

Frontend testing vs. monitoring

End-to-end testing with resource constraints is a great step to gain confidence and guarantee that your deployed code works no matter if your dependencies are slow or unavailable. But honestly, testing end-to-end is only the first step to real confidence because between today’s deployment and tomorrow plenty of other things can go wrong.

While end-to-end testing guarantees that your site works at one moment in time and helps to evaluate regressions in your source code, if you’re relying on third-party resources, your third-party provider’s downtime can still affect your site. And that’s sometimes unavoidable but at least, you should be aware of your dependencies’ effects on your site and bet on end-to-end monitoring. Be the first one to know when your vendors are struggling!

Conclusion

First, if you’re relying on resources that you don’t own, check how their unavailability could affect your frontend. Opt for self-hosting your assets, and if that’s impossible, rely on asynchronous and fail-safe implementations.

Second, test your site during deployments and stop, if new and risky dependencies were added to your frontend. A script element pointing to another CDN is quickly implemented. Be aware of your external resources and how they affect your application.

And third, if you have to rely on external resources, start monitoring your frontend, run your end-to-end tests on a schedule, and ensure that your site—and also those external resources!—are up and running. Don’t be me, coming to work only to learn that your customers have been looking at a blank screen 30 seconds long for the last 8 hours.

And maybe you want to give Checkly a try, we’ll run your Playwright tests for you. 😉

Playwright Tips From the Checkly Community

Stefan Judis — Wed, 01 Feb 2023 12:38:26 +0000

The Checkly community recently came together to talk Playwright in our public Slack! My fellow Playwright ambassador John Hill and I invited everyone to ask questions and share tips about Microsoft’s stellar end-to-end testing tool.

And there were plenty of them! Let’s have a look at my favorite questions (and learnings) coming from the community! Are you ready for a mixed bag of Playwright tricks? Let’s go!

How should you handle parallelism in your end-to-end tests?

In general, it’s best practice to parallelize as many tests as possible to avoid seeing your test duration going through the roof. No one likes to wait thirty minutes for a green deployment light and a test suite that takes forever (or flakiness) is the main reason your test efforts fail.

But as usual in software development, things are complicated and every complex project comes with its own challenges when it comes to testing.

A tricky case is test cases that involve resource updates. If all your tests run in parallel it’s easy to discover race conditions where one test creates or deletes a resource while another one is testing it, too! This problem is tricky to solve and there is no ideal solution but our CTO Tim Nolet shared a possible approach:

For non-destructive tests that are not relying on a state in your database, Tim recommends going all in with parallel testing. Your test cases could range from checking that a modal pops up after a button click or previews that are rendered when someone interacts with a text field. All these tests can run independently and in parallel.

With stateful tests, it’s a different story, though. If resource creation and updates are at play, it’s sometimes unavoidable to run tests sequentially to avoid one test messing with another one.

But how do you separate and run these different tests in your Playwright project?

To implement Tim’s recommendation in your project, you have to familiarize yourself with how Playwright handles parallelism.

In general, Playwright runs different test files in parallel, and all tests in a file are run sequentially. But watch out, the fullyParallel global config changes this behavior and runs every test case in parallel regardless of where it’s defined.

The default behavior lets you run tests that are in a single file sequential. For example, if you want to create, update and delete resources after another, having tests in a single file guarantees execution order.

That’s a good start, but unfortunately, it doesn’t guarantee that other spec files aren’t interfering with a similar resource and this again could lead to false positives. Luckily Playwright is highly configurable and you can disable parallelism via the CLI or Playwright config by defining the number of workers running your tests.

npx playwright test --workers=1

The global workers configuration enables you to run tests with different parallelization settings. Independent and stateless tests can be run fully parallel whereas sequential state-dependent tests can run in a sequence.

To split tests, leverage custom annotations or file path conventions to run Playwright in full parallelized or sequential mode. The example package.json below leverages different directories to separate parallel from sequential test cases.

{
  "name": "parallel",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test:parallel": "npx playwright test tests/parallel/",
    "test:sequence": "npx playwright test tests/sequence/ --workers=1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@playwright/test": "^1.29.2"
  }
}

With this setup, you can run npm run test:parallel and npm run:sequence without running into test conflicts. By separating the stateless and stateful tests, you’ll be able to reduce test collisions and run independent tests as quickly as possible.

But end-to-end tests that interact with the same resources are only one cause for flakiness. There are plenty of others!

What are common reasons for flaky tests and how can you avoid them?

There are many reasons for flaky tests. Giovanni Rago, our Head of Customer Solutions, shared his top flakiness offenders.

As Gio states, suboptimal waiting is a common problem in UI and end-to-end testing.

To test your application it has to have reached a particular state, JavaScript probably needs to be loaded or potential API calls had to resolve. There are multiple ways to go about testing your UIs.

But before going into explicit waiting scenarios, make sure to check Playwright’s auto-waiting mechanisms and web-first assertions. With all the built-in auto-waiting mechanisms you rarely need to worry about explicit waiting. For example, if you want to test a UI flow that includes a component that only becomes visible eventually, Playwright actions such as click wait for elements to become visible.

These features enable you to test highly asynchronous UIs with code that looks synchronous.

// Playwright retries to click this button until it works or times out
await page.getByRole('button').click();

But as Gio says, sometimes these built-in waiting mechanisms aren’t enough and you have to wait for an explicit event to happen or a UI state to be reached. John Hill takes things to another level and recommends watching specific network requests to make functionality deterministic.

If you click a button, that triggers an API call, you can wait for the API call to resolve and test if the UI reacted accordingly.

As John states:

If your browser has a HTTP API Backend, the #1 way to add determinism to your tests is to wait on the specific network requests to resolve.
99% of the flake I’ve experienced when testing a flaky frontend app comes from a flaky and unreliable backend.
The frontend tests are just the bearer of bad news.

If you want to check an example of this approach, here’s the code John and the team at Open MCT run to test UIs request-dependent at Nasa.

How can you emulate custom keyboard interactions in Playwright?

If you’re working on a SaaS product and aim for stellar UX, you might implement keyboard shortcuts and interactions. These could be challenging to test, but luckily, Playwright also supports a way to control a virtual keyboard.

Playwright’s keyboard functionality enables you to interact with input elements and pages like a real user.

await page.keyboard.press("Meta+KeyA")
await page.keyboard.type('This is a line of text\n');

This functionality is great for shortcuts but also valuable for highly custom and complex components. One example is a JS-based code editor like Monaco.

Millions of developers use Monaco because it’s built into VS Code (and app.checklyhq.com 😉). But how does it work under the hood and how would you test a custom editor if it’s embedded in your application?

Open the Monaco Playground to look at Microsoft’s open-source editor component. When you inspect the editor, you’ll see that to offer syntax highlighting, there are divs and spans all over the editor component. But how could you edit and test these and interact with the editor?

Monaco and other code editor components usually include a hidden or transparent textarea to catch keyboard events and make things more accessible. The textarea keystrokes are captured, canvas elements render code suggestions, and spans/divs display highlighted code. There’s a lot of magic involved and it’s pretty challenging to test.

Luckily, Playwright’s virtual keyboard helps out here and as the Playwright core team member Max Schmitt shared, page.keyboard is at your service to interact with Monaco.

await page.goto('https://microsoft.github.io/monaco-editor/playground.html');
const monacoEditor = page.locator(".monaco-editor").nth(0);
await monacoEditor.click();
await page.keyboard.press("Meta+KeyA");
await page.keyboard.type('This is a line of text\n');
await page.screenshot({ path: example.png });

This functionality allows testing and simulating keyboard interactions going beyond filling an input with a string value. Press specific key combinations, jump to the beginning of a line, or hit the “delete” key three times… It’s up to you to mimic your users’ keyboard behavior!

Related Reading: How we added custom languages, code completion and highlighting to the Monaco editor.

Real users are an unpredictable bunch, though. How can you guarantee that your application isn’t breaking when you “attack” it with your end-to-end tests?

How can you catch JavaScript errors in long end-to-end testing flows?

It’s always good to monitor if your application throws JavaScript exceptions when users interact with it. Luckily, reacting to JavaScript errors is straightforward in Playwright by leveraging emitted page events such as pageerror. React and count exceptions with a few lines of code.

// Log all uncaught errors to the terminal
page.on('pageerror', exception => {
  console.log(`Uncaught exception: "${exception}"`);
});

Related video: if you want to learn how these work, have a look at one of our Playwright tips on YouTube.

John Hill again puts things to the next level and shared how they implement JavaScript error tracking with Playwright fixtures. His approach allows you to collect all JS errors until the end of your test and only fail it then.

exports.test = base.test.extend({
   failOnConsoleError: [true, { option: true }],
   /**
    * Extends the base page class to enable console log error detection.
    * @see {@link https://github.com/microsoft/playwright/discussions/11690 Github Discussion}
    */
   page: async ({ page, failOnConsoleError }, use) => {
       // Capture any console errors during test execution
       const messages = [];
       page.on('console', (msg) => messages.push(msg));
       await use(page);

       // Assert against console errors during teardown
       if (failOnConsoleError) {
           messages.forEach(
               msg => expect.soft(msg.type(), `Console error detected: ${_consoleMessageToString(msg)}`).not.toEqual('error')
           );
       }
   }
});

The example leverages fixtures and soft assertions and if you’re curious, have a look at John’s provided example code.

Conclusion

These were my favorite Playwright learnings from our Community AMA session and I continue to be amazed at how far end-to-end testing has come. If you want to keep learning about Playwright, check out our Checkly YouTube channel where I share all these nitty-gritty tricks as I discover them. And if you have any questions, don’t hesitate to drop into Slack, I’m happy to help!

And lastly, you might know that testing is only the beginning of shipping stellar products. If you want to start monitoring your apps and be confident that your site works at all times, this is what we do at Checkly. Sign up and run Playwright tests at any time from around the world. It’s free and pretty cool, trust me! 😉

How low-level API calls can stabilize your end-to-end tests

Nico Domino — Tue, 21 Jun 2022 14:57:44 +0000

We're heavy end-to-end monitoring users here at Checkly and always experiment with how to architect our tests the best way. Over the past months, we've settled on a few workflows that make it much easier to spin up new tests, avoid code duplication, and make the entire test setup easier to manage.

One of those strategies is to strictly separate concerns in our tests.

For example, instead of having one check that creates and deletes a resource in the UI, we've split these actions into multiple test runs. One browser check tests the resource creation, and another one the resource deletion. But then, if one test is restricted to resource creation, when do you delete and clean up the resource? And also, if a test only checks the deletion, where do you create the resource in the first place?

This situation is when working on the API level helps. We introduced a custom API client to prepare and tear down all the UI test cases using the underlying API. A few lines of code speeded up the process and made tests more reliable.

We successfully use API calls in our UI tests to execute any prerequisite and postrequisite actions. Below we'll go into more detail on how we split up a large Playwright test codebase and how an API client helped us get there!

Sounds intriguing? Read on!

Plain API calls in end-to-end browser tests

Previously, our Playwright UI tests would create a Checkly dashboard and delete it all in one browser session. This structure led to issues where errors in the "create" phase would immediately cause the test to fail and not execute the test's "delete" portion. We littered our test account with dummy dashboards. To solve this problem, we split our "create" and "delete" tests into separate check runs.

But by decoupling these functionalities, we couldn't rely on a particular setup or clean-up while running the tests. For example, the "delete" test won't garbage collect the "create" test, and the "create" test won't generate entities for the "delete" test to remove.

What if we could work on the API directly to ensure our browser checks are running correctly without leaving traces behind? We introduced a custom API client to stabilize our Playwright tests.

Below is an excerpt of our "Delete Dashboard'' check. The test imports checklyApi from our simpleChecklyApiClient snippet and uses the API client to create a Checkly dashboard so that we can test its deletion in the UI.

This way, the check focuses on one functionality in a nice and clean way!

// DELETE DASHBOARD
const { chromium } = require("playwright")
const { checklyApi } = require("./snippets/simpleChecklyApiClient")

let dashboardId

;(async () => {
  try {
    // Create a dashboard via API, to be deleted right after
    dashboardId = await checklyApi.dashboards.create({
      customUrl: "<https://customdashboard.example.com>",
      header: "Dashboard Creation E2E Test",
    })

    const browser = await chromium.launch()
    const page = await browser.newPage()

    // Try to delete the dashboard using Playwright within a browser session
    // ...
    // ...
  } catch {
    console.error("Error deleting dashboard")
  } finally {
    // If anything fails in that process,
    // we explicitly delete the dashboard to ensure
    // this Check has no left-over side effects
    if (dashboardId) {
      checklyApi.dashboards.remove(dashboardId)
    }
  }
})()

But what happens if the end-to-end test fails to delete the dashboard?

We introduced a finally block to ensure that even if something goes wrong, we'll always clean up the test run by attempting to delete the dashboard with our API client.

A custom Node.js API wrapper

The checklyApi is a JavaScript class wrapper around axios. It includes methods for HTTP verbs that are preconfigured with our baseUrl and an API key. The object also comes with additional convenience methods that target explicit resources in our API (i.e. /checks, /check-groups, etc.). The client is easy to use and tailored to our test scenarios.

// CHECKLY API HELPER
const axios = require("axios")

const API_ROOT = process.env.E2E_API_ROOT_PRODUCTION
const API_TOKEN = process.env.E2E_CHECKLY_API_KEY_PRODUCTION
const ACCOUNT_ID = process.env.E2E_CHECKLY_ACCOUNT_ID_PRODUCTION

const headers = {
  Authorization: `Bearer ${API_TOKEN}`,
  "X-Checkly-Account": ACCOUNT_ID,
}

class ChecklyApi {
  #buildPath(url) {
    return `${API_ROOT}${url}`
  }

  #requestMethodWithNoData(method, url, config) {
    return axios({
      headers,
      ...config,
      method,
      url: this.#buildPath(url),
    })
  }

  #requestMethodWithData(method, url, data, config) {
    return axios({
      headers,
      ...config,
      method,
      url: this.#buildPath(url),
      data,
    })
  }

  async $get(url, config) {
    return (await this.#requestMethodWithNoData("get", url, config)).data
  }

  async $post(url, data, config) {
    return (await this.#requestMethodWithData("post", url, data, config)).data
  }

  // More convenience HTTP Methods

  dashboards = {
    create: (payload) => {
      return this.$post(`/v1/dashboards`, payload)
    },
    remove: (dashboardId) => {
      return this.$delete(`/v1/dashboards/${dashboardId}`)
    },
  }
}

module.exports = {
  checklyApi: new ChecklyApi(),
}

Having all the API client logic centralized in this helper class allows us to make adjustments in a single place and reuse the client across all of our checks. Like in the "Delete Dashboard" example check above, the usage is as simple as:

checklyApi.dashboards.remove(dashboardId)

And for completeness, here is the opposite check "Create Dashboard":

// CREATE DASHBOARD
const { chromium } = require("playwright")
const { checklyApi } = require("./snippets/simpleChecklyApiClient")

const dashboardUrl = "https://testdashboard.example.com"

;(async () => {
  let dashboardIdToDelete

  try {
    const browser = await chromium.launch()
    const page = await browser.newPage()

    // Navigate the UI and try to create a dashboard
    // ...
    // ...

    // Retrieve check ID from the creation response triggered by UI interactions
    const { dashboardId } = await (
      await page.waitForResponse((response) => {
        return (
          response.url().endsWith("/dashboards") &&
          response.request().method() === "POST" &&
          response.status() === 200
        )
      })
    ).json()
    dashboardIdToDelete = dashboardId

    // Dashboard should be visible in the list
    await page.locator(`a[href="${dashboardUrl}"]`).waitFor()
  } catch (e) {
    console.error(e)
  } finally {
    if (dashboardIdToDelete) {
      // Housekeeping, delete the created dashboard via API
      checklyApi.dashboards.remove(dashboardId)
    }
  }
})()

Both Playwright tests leverage plain API calls to set up and tear down, but there's a crucial difference.

How to extract resource ids using Playwright request interception

Combining API calls with end-to-end testing can sometimes be challenging. Setup steps are generally easier to realize than teardowns. But why's that?

When you create resources on the API level, you're in control of the data, requests and responses. But if you want to clean up and delete resources created in a browser session, this data might not be available to you.

This is when Playwright request interceptors shine!

// Retrieve check ID from the creation response
const { dashboardId } = await (
  await page.waitForResponse((response) => {
    return (
      response.url().endsWith("/dashboards") &&
      response.request().method() === "POST" &&
      response.status() === 200
    )
  })
).json()

We run our UI tests as usual and listen to our app requests using waitForResponse to evaluate what resources need to be deleted in the teardown phase. By monitoring the network activity, we can snatch all the required resource ids and delete anything created in your Playwright test using the API.

There are no traces, no matter if the UI test fails or succeeds!

Wrapping Up

Writing fast and stable end-to-end tests can be a real challenge, but keeping tests lean and separating them into smaller chunks helps us to keep our test coverage high without slowing us down.

Our tests are now easier to debug, they share more code, and everything runs faster and in parallel. And even though making API calls in our end-to-end tests felt odd initially, we're not creating sequential test waterfalls anymore, thanks to a tiny Node.js API wrapper. Most of our Playwright tests are idempotent, so that we can run them continuously. And the best part: we're sure everything's cleaned up, no matter how many times we run our tests. This is how it should be!

This practice is only the tip of the iceberg. We're continuously improving our monitoring setup and will share more tricks here on the blog. If you want to learn more about end-to-end monitoring, we've plenty of other resources available. Until then, happy testing!

Avoiding hard waits in Playwright and Puppeteer

Tim Nolet 👨🏻‍🚀 — Thu, 16 Dec 2021 15:39:13 +0000

Looking to solve the issue of a page or element not being loaded, many take the shortcut of waiting for a fixed amount of time - adding a hard wait, in other words. This is regarded as an anti-pattern, as it lowers performance and increases the chances of a script breaking (possibly intermittently). Let's explore how those issues arise and what better solutions we can use to avoid them.

The problems with hard waits

Hard waits do one thing and one thing only: wait for the specified amount of time. There is nothing more to them. This makes them dangerous: they are intuitive enough to be favoured by beginners and inflexible enough to create serious issues.

Let's explore these issues in practical terms through an example. Imagine the following situation: our script is running using a tool without any sort of built-in smart waiting, and we need to wait until an element appears on a page and then attempt to click it. We try to solve this issue with a hard wait, like Puppeteer's page.waitFor(timeout).

This could looks something like the following:

await page.waitFor(1000); // hard wait for 1000ms
await page.click('#button-login');

In such a situation, the following can happen:

1) We can end up waiting for a shorter amount of time than the element takes to load!

In this case, our hard wait terminates and our click action is attempted too early. The script terminates with an error, possibly of the "Element not found" sort.

2) The element can load before our hard wait has expired.

While the element is correctly clicked once our wait expires, and our script continues executing as planned, we are wasting precious time - likely on each hard wait we perform. Across multiple scripts and suites, this can add up to noticeable drag on build time.

In general, with hard waits we are virtually always waiting too little or too long. In the worst case scenario, the fluctuations in load time between different script executions are enough to make the wait sometimes too long and sometimes too short (meaning we will switch between scenario 1 and 2 from above in an unpredictable manner), making our script fail intermittently. That will result in unpredictable, seemingly random failures, also known as flakiness.

Flakiness, a higher-than-acceptable false failure rate, can be a major problem. It is essentially a source of noise, making it harder to understand what the state of the system we are testing or monitoring really is. Not only that, but stakeholders who routinely need to investigate failures only to find out that they are script-related (instead of system-related) will rapidly lose confidence in an automation setup.

How to fix it

To avoid these issues, we have to ditch hard waits completely outside debugging scenarios. That means that hard waits should never appear in production scripts under any circumstance.

Our aim should be to wait just long enough for the element to appear. We want to always be certain the element is available, and never waste any time doing that. Luckily most automation tools and frameworks today offer multiple ways to achieve this. We can call these "smart waits".

Different tools approach the broad topic of waiting in different ways. Both Puppeteer and Playwright offer many different kinds of smart waits, but Playwright takes things one step further and introduces an auto-waiting mechanism on most page interactions.

Let's take a look at different smart waiting techniques and how they are used.

Built-in waits

Playwright comes with built-in waiting mechanisms on navigation and page interactions. Since these are baked into the tool itself, it is good to get familiar with the logic behind them, as well as how to override the default behaviour when necessary.

Explicit waits

Explicit waits are a type of smart wait we invoke explicitly as part of our script. We will want to use them more or less often depending on whether our automation tool has a built-in waiting mechanism (e.g. Playwright) or requires us to handle all the waiting (e.g. Puppeteer).

If you can rely on automatic waits, use explicit waits only when necessary. An auto-wait system failing once is no good reason for ditching the approach completely and adding explicit waits before every page load and element interaction. If the tool you are using does not do auto-waiting, you will be using explicit waits quite heavily (possibly after each navigation and before each element interaction), and that is fine - there is just less work being done behind the scenes, and you are therefore expected to take more control into your hands.

Waiting on navigations and network conditions

On a page load, we can use the following:

page.waitForNavigation to wait until a page navigation (new URL or page reload) has completed.
page.waitForLoadState for Playwright, waits until the required load state has been reached (defaults to load); page.waitForNetworkIdle with Puppeteer, a narrower method to wait until all network calls have ended.
page.waitForURL with Playwright, waits until a navigation to the target URL.

All the above default to waiting for the load event, but can also be set to wait for:

the DOMContentLoaded event.
Playwright only: networkidle, raised when there are no network connections for at least 500 ms.
Playwright only: commit, when the network response is received and the document starts loading (Playwright only).
Puppeteer only: networkidle0, raised when there are no network connections for at least 500 ms.
Puppeteer only: networkidle2, raise when the there are no more than 2 network connections for at least 500 ms.

Lazy-loaded pages might require extra attention when waiting for the content to load, often demanding explicitly waiting for specific UI elements. See the following section.

Additionally, we can also wait until a specific request is sent out or a specific response is received with page.waitForRequest and page.waitForResponse. These two methods are key for implementing request and response interception.

Waiting for an element

We can also explicitly wait for a specific element to appear on the page. This is normally done via page.waitForSelector or a similar method, like page.waitForXPath (Puppeteer only). A good knowledge of selectors is key to enable us to select precisely the element we need to wait for.

Waiting on page events

With Playwright, we can also directly wait on page events using page.waitForEvent.

Waiting on page functions

For more advanced cases, we can pass a function to be evaluated within the browser context via page.waitForFunction.

Takeaways

Never use hard waits outside of debugging
Use smart waits instead, choosing the best one for your situation
Use more or less smart waits depending on whether your tool support auto-waits

Banner image: detail from "IMG_0952" by sean_emmett is licensed under CC BY-NC-SA 2.0

Migrating from Puppeteer to Playwright

Tim Nolet 👨🏻‍🚀 — Tue, 16 Nov 2021 17:13:26 +0000

This article originally appeared in Checkly's Guides.

Puppeteer and Playwright today

While they share a number of similarities, Puppeteer and Playwright have evolved at different speeds over the last two years, with Playwright gaining a lot of momentum and arguably even leaving Puppeteer behind.

These developments have led many to switch from Puppeteer to Playwright. This guide aims to show what practical steps are necessary and what new possibilities this transition enables. Do not let the length of this article discourage you - in most cases the migration is quick and painless.

Why switch

While a comprehensive comparison of each tool's strengths and weaknesses could fill up a guide of its own (see our previous benchmarks for an example), the case for migrating to Playwright today is rather straightforward:

As of the writing of this guide, Playwright has been frequently and consistently adding game changing features (see below for a partial list) for many months, with Puppeteer releasing in turn mostly smaller changes and bug fixes. This led to a reversal of the feature gap that had once separated the two tools.
Playwright maintains an edge in performance in real-world E2E scenarios (see benchmark linked above), resulting in lower execution times for test suites and faster monitoring checks.
Playwright scripts seem to run even more stable than their already reliable Puppeteer counterparts.
The Playwright community on GitHub, Twitter, Slack and beyond has gotten very vibrant, while Puppeteer's has gone more and more quiet.

What to change in your scripts - short version

Below you can find a cheat sheet with Puppeteer commands and the corresponding evolution in Playwright. Keep reading for a longer, more in-depth explanation of each change.

Remember to add await as necessary.

Puppeteer	Playwright
`require('puppeteer')`	`require('playwright')`
`puppeteer.launch(...)`	`playwright.chromium.launch(...)`
`browser.createIncognitoBrowserContext(...)`	`browser.newContext(...)`
`page.setViewport(...)`	`page.setViewportSize(...)`
`page.waitForSelector(selector)` `page.click(selector);`	`page.click(selector)`
`page.waitForXPath(XPathSelector)`	`page.waitForSelector(XPathSelector)`
`page.$x(xpath_selector)`	`page.$(xpath_selector)`
`page.waitForNetworkIdle(...)`	`page.waitForLoadState({ state: 'networkidle' }})`
`page.waitForFileChooser(...)`	Removed, handled differently.
`page.waitFor(timeout)`	`page.waitForTimeout(timeout)`
`page.type(selector, text)`	`page.fill(selector, text)`
`page.cookies([...urls])`	`browserContext.cookies([urls])`
`page.deleteCookie(...cookies)`	`browserContext.clearCookies()`
`page.setCookie(...cookies)`	`browserContext.addCookies(cookies)`
`page.on('request', ...)`	Handled through page.route.
`elementHandle.uploadFile(...)`	`elementHandle.setInputFiles(...)`
Tricky file download.	Better support for downloads.

Did we forget anything? Please let us know by getting in touch, or submit your own PR.

What to change in your scripts - in depth

Require Playwright package

In Puppeteer, the first few lines of your script would have most likely looked close to the following:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // ...

With Playwright you do not need to change much:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // ...

Playwright offers cross-browser support out of the box, and you can choose which browser to run with just by changing the first line, e.g. to const { webkit } = require('playwright');
In Puppeteer, this would have been done throught the browser's launch options:

  const browser = await puppeteer.launch({ product: 'firefox' })

The browser context

Browser contexts already existed in Puppeteer:

const browser = await puppeteer.launch();
const context = await browser.createIncognitoBrowserContext();
const page = await context.newPage();

Playwright's API puts even more importance on them, and handles them a little differently:

const browser = await chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();

Like in Puppeteer, for basic cases and single-page flows, you can use the default context:

const browser = await chromium.launch();
const page = await browser.newPage();

When in doubt, explicitly create a new context at the beginning of your script.

Waiting

The auto-waiting mechanism in Playwright means you will likely not need to care about explicitly waiting as often. Still, waiting being one of the trickiest bits of UI automation, you will still want to know different ways of having your script explicitly wait for one or more conditions to be met.

In this area, Playwright brings about several changes you want to be mindful of:

page.waitForNavigation and page.waitForSelector remain, but in many cases will not be necessary due to auto-waiting.
page.waitForEvent has been added.
Puppeteer's page.waitForXPath has been incorporated into page.waitForSelector, which recognises XPath expressions automatically.
page.waitForFileChooser been removed removed (see the official dedicated page and our file upload example for new usage)
page.waitForNetworkIdle has been generalised into page.waitForLoadState (see the networkidle state to recreate previous behaviour)
page.waitForUrl has been added allowing you to wait until a URL has been loaded by the page's main frame.
page.waitFor(timeout) becomes page.waitForTimeout(timeout).

This is as good a place as any to remind that page.waitForTimeout should never be used in production scripts! Hard waits/sleeps should be used only for debugging purposes.

Setting viewport

Puppeteer's page.setViewport becomes page.setViewportSize in Playwright.

Typing

While puppeteer's page.type is available in Playwright and still handles fine-grained keyboard events, Playwright adds page.fill specifically for filling and clearing forms.

Cookies

With Puppeteer cookies are handled at the page level; with Playwright you manipulate them at the BrowserContext level.

The old...

...become:

Note the slight differences in the methods and how the cookies are passed to them.

XPath selectors

XPath selectors starting with // or .. are automatically recognised by Playwright, whereas Puppeteer had dedicated methods for them. That means you can use e.g. page.$(xpath_selector) instead of page.$x(xpath_selector), and page.waitForSelector(xpath_selector) instead of page.waitForXPath(xpath_selector). The same holds true for page.click and page.fill.

Device emulation

Playwright device emulation settings are set at Browser Context level, e.g.:

const pixel2 = devices['Pixel 2'];
const context = await browser.newContext({
  ...pixel2,
});

On top of that, permission, geolocation and other device parameters are also available for you to control.

File download

Trying to download files in Puppeteer in headless mode can be tricky. Playwright makes this more streamlined:

const [download] = await Promise.all([
  page.waitForEvent('download'),
  page.click('#orders > ul > li:nth-child(1) > a')
])

const path = await download.path();

See our example on file download.

File upload

Puppeteer's elementHandle.uploadFile becomes elementHandle.setInputFiles.

See our example on file upload.

Request interception

Request interception in Puppeteer is handled via page.on('request', ...):

await page.setRequestInterception(true)

page.on('request', (request) => {
  if (request.resourceType() === 'image') request.abort()
  else request.continue()
})

In Playwright, page.route can be used to intercept requests with a URL matching a specific pattern:

await page.route('**/*', (route) => {
  return route.request().resourceType() === 'image'
    ? route.abort()
    : route.continue()
})

See our full guide on request interception for more examples.

For many of the points in the list above, variations of the same function exist at Page, Frame and ElementHandle level. For simplicity, we reported only one.

New possibilities to be aware of

When moving from Puppeteer to Playwright, make sure you inform yourself about the many completely new features Playwright introduces, as they might open up new solutions and possibilities for your testing or monitoring setup.

New selector engines

Playwright brings with it added flexibility when referencing UI elements via selectors by exposing different selector engines. Aside from CSS and XPath, it adds:

Playwright-specific selectors, e.g.: :nth-match(:text("Buy"), 3)
Text selectors, e.g.: text=Add to Cart
Chained selectors, e.g.: css=preview >> text=In stock

You can even create your own custom selector engine.

For more information on selectors and how to use them, see our dedicated guide.

Saving and reusing state

Playwright makes it easy for you to save the authenticated state (cookies and localStorage) of a given session and reuse it for subsequent script runs.

Reusing state can save significant amounts of time on larger suites by skipping the pre-authentication phase in scripts where it is not supposed to be directly tested / monitored.

Locator API

You might be interested in checking out Playwright's Locator API, which encapsulates the logic necessary to retrieve a given element, allowing you to easily retrieve an up-to-date DOM element at different points in time in your script.

This is particularly helpful if you are structuring your setup according to the Page Object Model, or if you are interested to do start doing that.

Playwright Inspector

The Playwright Inspector is a GUI tool that comes in very handy when debugging scripts, allowing you to step instruction-by-instruction through your script to more esily identify the cause of a failure.

The Inspector also comes in handy due its ability to suggest selectors for page elements and even record new scripts from scratch.

Playwright Test

Playwright comes with its own runner, Playwright Test, which adds useful features around end-to-end testing, like out-of-the-box parallelisation, test fixtures, hooks and more.

Trace Viewer

The Playwright Trace Viewer allows you to explore traces recorded using Playwright Test or the BrowserContext Tracing API. Traces are where you can get the most fine-grained insights into your script's execution.

Test Generator

You can use the Playwright Test Generator to record interactions in your browser. The output will be a full-fledged script ready to review and execute.

Switching to Playwright for richer browser check results

Checkly users switching to Playwright can take advantage of its new Rich Browser Check Results, which come with tracing and Web Vitals and make it easier to isolate the root cause of a failed check and remediate faster.

This reveals additional information about the check execution, including:

Overview of all errors raised (console, network and script errors)
A timeline summarising the execution across page navigations
For each page visited, a network & timing timeline, Web Vitals, console and network tabs.
In case of a failing check, a screenshot on failure.

Aside from running a Playwright script, performance and error tracing also require the use of Runtime 2021.06 or newer.

Note that cross-browser support is not available on Checkly - our Browser checks run on Chromium only.

banner image: "rocket engine" by industrial arts, licensed under CC BY 2.0

Debugging Puppeteer & Playwright scripts

Tim Nolet 👨🏻‍🚀 — Tue, 10 Aug 2021 14:27:03 +0000

This article originally appeared on Checkly's Learn Puppeteer & Playwright

Understanding why a script does not work as expected, or just what the root cause of a failure is, is a key skill for automation. Given its importance and its sometimes deceptive complexity, debugging is a topic that should receive quite some attention.

This article will explore basic concepts and tools to point beginners in the right direction.

Awareness comes first

Script debugging is firstly about observing and understanding. Finding out what is causing the failure (or misbehaviour) in your execution heavily depends on your knowledge of:

What the script you are looking at is supposed to do
How the application the script is running against is supposed to behave at each step of the script

When approaching a debugging session, make sure the above points are taken care of. Skipping this step is way more likely to cost you additional time than it is to save you any.

The error message

Error messages are not present in every scenario: we might be trying to understand why a script passes, or why it takes longer than expected. But when we have access to an error message, we can use it to guide us.

The error, in and of its own, is not always enough to understand what is going wrong with your script. Oftentimes, there can be multiple degrees of separation between the error and its root cause. For example: an "Element not found" error might be alerting you to the fact that an element is not being found on the page, but that itself might be because the browser was made to load the wrong URL in the first place.

Do not fall into the easy trap of reading the error message and immediately jumping to conclusions. Rather, take the error message, research it if needed, combine it with your knowledge of script and app under test and treat it as the first piece to the puzzle, rather than the point of arrival of your investigation.

Good knowledge of the automation tool you are using will also help add more context to the error message itself.

Gaining visibility

Given that Headless browser scripts will run without a GUI, a visual assessment of the state of the application needs additional steps.

One possibility is to adding screenshots in specific parts of the script, to validate our assumptions on what might be happening at a given moment of the execution. For example, before and after a problematic click or page transition:

For Playwright:

...
await page.screenshot({ path: 'before_click.png' });
await page.click('#button')
await page.screenshot({ path: 'after_click.png' });
...

For Puppeteer:

...
await page.screenshot({ path: 'before_click.png' });
await page.waitForSelector('#button');
await page.click('#button')
await page.screenshot({ path: 'after_click.png' });
...

Another way to better observe our script's execution is to run in headful mode:

For Playwright:

...
const browser = await chromium.launch({ headless: false, slowMo: 20 });
...

For Puppeteer:

...
const browser = await puppeteer.launch({ headless: false, slowMo: 20 });
...

We can then tweak the slowMo option, which adds a delay in milliseconds between interactions, to make sure the execution is not too fast for us to follow.

Increasing logging

Sometimes we need to try and see the execution through our automation tool's eyes. Added logging can help by taking us step-by-step through every command as it is executed.

For Playwright:

DEBUG=pw:api node script.js

For Puppeteer:

DEBUG="puppeteer:*" node script.js

Accessing DevTools

A wealth of information is available through the Chrome Developer Tools. We can configure our browser to start with the DevTools tab already open (this will automatically disable headless mode), which can be helpful when something is not working as expected. Careful inspection of the Console, Network and other tabs can reveal hidden errors and other important findings.

For Playwright:

...
await chromium.launch({ devtools: true });
...

For Puppeteer:

...
await browser.launch({ devtools: true });
...

We can also use the console to directly try out a selector on the page in its current state, e.g. with document.querySelector or document.querySelectorAll.

If we are using Playwright, we can also run in debug mode with PWDEBUG=console node script.js. This provisions a playwright object in the browser which allows us to also try out Playwright-specific selectors.

The Playwright Inspector

The Playwright Inspector is a GUI tool which exposes additional debugging functionality, and can be launched using PWDEBUG=1 npm run test.

The Inspector allows us to easily step through each instruction of our script, while giving us clear information on the duration, outcome, and functioning of each. This can be helpful in getting to the root cause of some of the more generic errors.

The Inspector includes additional handy features such as selector generation and debugging, as well as script recording.

OpenAPI/Swagger Monitoring

Tim Nolet 👨🏻‍🚀 — Wed, 28 Apr 2021 14:04:24 +0000

OpenAPI and Swagger help users design and document APIs in a way that is readable from both humans and machines. As a consequence, they can also be used to generate the code that will run the specified API - both on the provider and consumer side. Can we leverage this same principle to simplify API monitoring? After a brief first look at OpenAPI and Swagger, this article will show how we can quickly use them to monitor a new or existing API.

The OpenAPI Specification

The OpenAPI Specification (OAS) specifies a standard, language-agnostic and machine-readable format to describe a web API in one or more files. Nowadays, it acts as a vendor-neutral standard for describing the structure and behaviour of HTTP-based APIs, and exists as part of the OpenAPI Initiative (OAI).

Following the OAS brings the following advantages:

A description of the API that is readable from both humans and machines.
The possibility of automatically generating documentation, implementation logic and even mock servers for testing.
Early validation of the data flows happening within the API.

For those new to the OAS and wanting to get a basic understanding without diving straight into the specification itself, the official OpenAPI Specification Explained guide is a great place to start.

Swagger vs OpenAPI

The OpenAPI Specification is based on the SmartBear SoftwareSmartBear Software, which had previously been a project driven by SmartBear Software. SmartBear donated the Swagger Specification to the Linux Foundation in 2015. In a sense, the OAS is the newer, fully open source-driven incarnation of the Swagger Specification.

The name Swagger today indicates a set of tools, both free and paid, that support users of the OpenAPI ecosystem.

Exploring the specification

The core of our API specification will happen in the OpenAPI root document, normally named openapi.json or openapi.yml. We can choose between JSON and YAML formats as we like.

The bare minimum we need to declare is the OpenAPI version, the basic information about our API, and finally the available endpoints.

openapi: 3.0.0
info:
  title: OpenAPI document for our new Payment API
  version: 0.0.1
paths: {} # Nothing here yet

An API without endpoints is not very useful. We can add our first, together with a GET operation:

openapi: 3.0.0
info:
  title: OpenAPI document for our new Payment API
  version: 0.0.2
paths:
  # User account balance
  /balance/{id}:
    # Retrieve the user's balance on the given account
    get: 
      ...

The next step is to define what the operation will look like. In this case, we will specify some constraints on the id parameter, as well as the response.

openapi: 3.0.0
info:
  title: OpenAPI document for our new Payment API
  version: 0.0.3
paths:
  # User account balance
  /balance/{id}:
    # Retrieve the user's balance on the given account
    get: 
      summary: Get the user's account balance
      description: Retrieves the current account balance for the given user.
      operationId: get-balance
      parameters:
      - name: id
        in: path
        required: true
        schema:
          type: string
          format: uuid
      responses:
        "200":
          description: "OK"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/balance"

Notice we are explicitly defining which schemas the parameter and response should conform to. In the case of our response, we are referring to an
externally defined schema. This is helpful when we need to reuse a schema definition multiple times across our spec file.

As we proceed, we might add multiple endpoints, each with one or more operations.

openapi: 3.0.0
info:
  title: OpenAPI document for our new Payment API
  version: 0.0.4
paths:
  # User account balance
  /balance/{id}:
    # Retrieve the user's balance on the given account
    get:
      summary: Get the user's account balance
      ...
  /transactions:
    get:
      summary: Retrieve transaction
      ...
    post:
      summary: Submit transaction
      ...
  ...

We can keep on building out our API spec as needed. The more precisely we describe our API in this file, the more useful this specification will become, whether we use it for documentation or code generation purposes.

Generating documentation from OpenAPI

Let's now take a look at a more complex, finished example: the Swagger PetStore demo. Even if we had written it ourselves, this description of a complete API could be overwhelming if we just tried to parse the file line by line. To first-time users, that could feel even more daunting. A better option would be to use the open-source toolset Swagger offers to generate human-readable, interactive API documentation from the file we already have.

Pasting the content of the spec file to Swagger Editor will produce a preview of our Swagger UI documentation. This is helpful for consumers of the API, who will be presented with an orderly documentation page breaking down each endpoint while also allowing users to test out different operations.

Generating boilerplate code from OpenAPI

When writing a new API, we will oftentimes need to produce a certain amount of boilerplate code for both our provider (e.g. a server exposing our API for consumption) and consumer (e.g. an SDK for our API) applications. Once we have our OpenAPI file in place, we can use Swagger Codegen (also available within Swagger Editor) to automatically generate that code for us, saving us precious time and reducing the avenues for human error.

The following snippet is an example of the code Swagger Codegen can generate starting from an OAS file.

/**
   * Invokes the REST service using the supplied settings and parameters.
   * @param {String} path The base URL to invoke.
   * @param {String} httpMethod The HTTP method to use.
   * @param {Object.<String, String>} pathParams A map of path parameters and their values.
   * @param {Object.<String, Object>} queryParams A map of query parameters and their values.
   * @param {Object.<String, Object>} collectionQueryParams A map of collection query parameters and their values.
   * @param {Object.<String, Object>} headerParams A map of header parameters and their values.
   * @param {Object.<String, Object>} formParams A map of form parameters and their values.
   * @param {Object} bodyParam The value to pass as the request body.
   * @param {Array.<String>} authNames An array of authentication type names.
   * @param {Array.<String>} contentTypes An array of request MIME types.
   * @param {Array.<String>} accepts An array of acceptable response MIME types.
   * @param {(String|Array|ObjectFunction)} returnType The required type to return; can be a string for simple types or the
   * constructor for a complex type.
   * @param {module:ApiClient~callApiCallback} callback The callback function.
   * @returns {Object} The SuperAgent request object.
   */
  exports.prototype.callApi = function callApi(path, httpMethod, pathParams,
      queryParams, collectionQueryParams, headerParams, formParams, bodyParam, authNames, contentTypes, accepts,
      returnType, callback) {

    var _this = this;
    var url = this.buildUrl(path, pathParams);
    var request = superagent(httpMethod, url);

    // apply authentications
    this.applyAuthToRequest(request, authNames);

    // set collection query parameters
    for (var key in collectionQueryParams) {
      if (collectionQueryParams.hasOwnProperty(key)) {
        var param = collectionQueryParams[key];
        if (param.collectionFormat === 'csv') {
          // SuperAgent normally percent-encodes all reserved characters in a query parameter. However,
          // commas are used as delimiters for the 'csv' collectionFormat so they must not be encoded. We
          // must therefore construct and encode 'csv' collection query parameters manually.
          if (param.value != null) {
            var value = param.value.map(this.paramToString).map(encodeURIComponent).join(',');
            request.query(encodeURIComponent(key) + "=" + value);
          }
        } else {
          // All other collection query parameters should be treated as ordinary query parameters.
          queryParams[key] = this.buildCollectionParam(param.value, param.collectionFormat);
        }
      }
    }

    // set query parameters
    if (httpMethod.toUpperCase() === 'GET' && this.cache === false) {
        queryParams['_'] = new Date().getTime();
    }
    request.query(this.normalizeParams(queryParams));

  ...

Depending on how much is described in a given API spec, it is possible to move even further and generate tests, mock servers and more. See openapi.tools for an up-to-date list of tools belonging to the OpenAPI ecosystem.

Generated monitoring checks with Checkly

API monitoring is key to ensure that our endpoints are returning the correct results in an acceptable timeframe. If we can generate APIs from a OAS description file, we can also generate monitoring checks for them. Checkly is a member of the OpenAPI initiative, and allows us to import an OpenAPI spec file and automatically creates one or more checks depending on the amount of information available. All it takes is a couple of clicks.

When creating an API check, select the import from Swagger / OpenAPI button.

That will bring up the import dialog. This is where we give Checkly the URL to the Swagger/OpenAPI specification file.

Before anything is created on our behalf, Checkly will show us a summary of all the checks that will be imported. We can choose to import them all defined checks or just a subset.

Notice that we can also select whether to import headers, query params and other settings or just to ignore them for now.

Once we confirm, the checks will be created for us and will start running according to the schedule we have selected.

Based on how comprehensively the file we imported described the API we want to monitor, we might have to tweak certain checks manually before they are fully set up.

This usually includes adding missing headers or body contents that might not have been explicitly defined in the imported file.

We can open the newly created checks and change any of their settings.

After a small amount of tinkering, or none at all, we should have all our checks up and running.

For APIs medium and large, and for any that is already described with an OAS file, this check-generating procedure can:

Save us a large amount of time otherwise spent in manual check setup.
Reduce the chance for human error in check configuration.
Help us ensure the right checks are set up according to our single source of truth.

Higher check coverage can ultimately give us higher confidence that our API is working as expected, and basing our monitoring on our OpenAPI specification enables us to get to higher coverage faster - without losing flexibility when building our checks.

Building on our generated monitors

As we look to increase our endpoint coverage across our APIs, we might want to explore the following resources:

Learning more about API monitoring basics and best practices.
Defining our API checks as code to scale our setup.
Integrating our API monitoring with E2E monitoring for our websites.

API Monitoring for the JAMStack

Tim Nolet 👨🏻‍🚀 — Fri, 09 Apr 2021 13:51:51 +0000

Originally published at https://www.checklyhq.com/guides/api-monitoring

Application Programming Interfaces (APIs) are used throughout software to define interactions between different software applications. In this article we focus on web APIs specifically, taking a look at how they fit in the JAMStack architecture and how we can set up API monitoring in order to make sure they don't break and respond fast.

APIs and the JAMStack

With the rise of the JAMStack, the already broadly used web APIs have been brought further into the spotlight and explicitly named as cornerstone of a new way of building web applications. In the JAMStack paradigm, applications rely on APIs (the A in "JAM") returning structured data (JSON or XML) when queried via the HTML and Javascript-based frontend.

The API calls might be aimed at internal services or at third-parties handling complex flows such as content management, authentication, merchant services and more. An example of third-party API could be Stripe, which acts as payment infrastructure for a multitude of businesses.

Given their importance in this new kind of web application, APIs both internal and external need to be tightly monitored, as failures and performance degradations will immediately be felt by the end-user.

API failures

API endpoints can break in a variety of ways. The most obvious examples are:

The endpoint is unresponsive/unreachable.
The response is incorrect.
The response time is too high.

All of the above can result in the application becoming broken for the end-user. This applies to internal APIs and, especially in the case of JAMStack applications, to third parties as well. API checks allow us to monitor both by mimicking the end-user's own behaviour.

API checks

If we were interested in just verifying a server or a virtual machine's availability, we could rely on a simple ping/uptime monitoring solution. API monitoring is more fine-grained than that though, as we need to validate functionality and performance on each API endpoint. API checks do exactly that, and they are composed of the following:

An HTTP request.
One or more assertions, used to specify exactly what the response should look like, and fail the check if the criteria are not met.
A threshold indicating the maximum acceptable response time.

The more customisable the HTTP request is, the more cases can be covered, for example with authentication, headers and payloads.

It is worth noting that in real-world scenarios, requests do not happen in a vacuum: they are often handling data retrieved previously, possibly by earlier API calls. Therefore, some mechanism to gather this data and inject it into the request is often needed.

Let's dive in deeper into each point.

Configurable HTTP request

There is a large variety of valid requests that a user might make to a given endpoint. Being able to customise all aspects of our test request is therefore fundamental. Key aspects are:

Method, like GET, PUT, POST, DELETE, etc
Headers, like Accept, Authorization, Content-Type, Cookie, User-Agent, etc
Query parameters

Essentially, we are trying to craft a complete request for exact endpoint. Not only that, but we might want to have multiple requests set up to cover specific options or negative cases, too.

One such case can be where user-specified parameters such as pagination and timeframes might largely change the response. This is exemplified by the List Customers method in Stripe's Customer API, which we can use to query elements in very different ways, such as by just specifying a limit of results or asking for all results linked to a specific creation date. In this case, both of the following cases are worth monitoring:

curl https://api.stripe.com/v1/customers \
  -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \
  -d limit=3 \
  -G

curl https://api.stripe.com/v1/customers \
  -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \
  -d created=1616519668 \
  -G

If we chose to set up a call using Javascript, for example, we could achieve the same call as in the first case above using axios:

const { default: axios } = require("axios");
const AUTH_TOKEN = Buffer.from(process.env.API_KEY).toString('base64')

axios({
    method: 'get',
    url: 'https://api.stripe.com/v1/customers',
    headers: { 
        'Authorization': `Basic ${AUTH_TOKEN}`,
        'content-type': 'application/x-www-form-urlencoded'
    },
    data: 'limit=3'
}).then((response)=> {
    console.log(response.data)
})

Assertions

To validate the API response, we should be able to check against

Status code
Headers
Body

Let's look at an example: creating a customer via the Stripe Customer API. Since we are not the API's developers, we are assuming the result we get running call right now is correct and can be used to model our assertions. Let's run the following curl command in verbose mode:

curl -v https://api.stripe.com/v1/customers \
  -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \
  -d description="My First Test Customer (created for API docs)"

Within the lengthy output we find the respose (in curl denoted by the '<' symbol), and in it all the important details we need for our assertions.

First, we notice the successful status code:

< HTTP/2 200

After that, we can see the headers, which we might want to check for:

< content-type: application/json
< content-length: 1190
< access-control-allow-credentials: true
< access-control-allow-methods: GET, POST, HEAD, OPTIONS, DELETE
< access-control-allow-origin: *
< access-control-expose-headers: Request-Id, Stripe-Manage-Version, X-Stripe-External-Auth-Required, X-Stripe-Privileged-Session-Required
< access-control-max-age: 300
< cache-control: no-cache, no-store
< request-id: req_S9P5NqvZXzvvS0
< stripe-version: 2019-02-19
< x-stripe-c-cost: 0
< strict-transport-security: max-age=31556926; includeSubDomains; preload

And finally the response body, which we might want to inspect to ensure the right data is being sent back:

{ 
  "id": "cus_JAp37QquOLWbRs",
  "object": "customer",
  "account_balance": 0,
  "address": null,
  "balance": 0,
  "created": 1616579618,
  [clipped]

We could expand on our previous code example by add adding an assertion library, such as chai's or Jest expect:

const { default: axios } = require("axios");
const expect = require('expect')

const AUTH_TOKEN = Buffer.from(process.env.API_KEY).toString('base64')

axios({
    method: 'get',
    url: 'https://api.stripe.com/v1/customers',
    headers: { 
        'Authorization': `Basic ${AUTH_TOKEN}`,
        'content-type': 'application/x-www-form-urlencoded'
    },
    data: 'limit=3'
}).then((response)=> {
    console.log(response.data)
    expect(response.status).toBe(200) // 1) assert again status code 
    expect(response.headers['content-type']).toBe('application/json') // 2) assert against header
    expect(response.data['has_more']).toBe(true) // 3) assert against body
})

We are now asserting against all three point mentioned above. We could of course go on and add additional assertions against both headers and body.

Response time thresholds

Having an endpoint return the correct result is only half the battle. It is imperative that the response reaches the user quickly enough not to disrupt any dependent workflow. In the worst case, where the response time exceeds what the end user is prepared to wait, a performance failure is undistinguishable from a functional one.

The easiest way to handle this requirement would be to assert that the specific response time be lower than a certain value, or even just set a timeout for our axios request by adding the timeout: 7500 property in the previously shown request config.

Instead of simply asserting against a specific response, we might want to set different thresholds: based on the nature of our service, a 2x slowdown might still leave it in what we define as an operational state, while a 10x one might not.

API monitoring best practices

Now that we are clear on the key requirements for setting up API checks, let's think about what and how we should monitor.

Monitor every endpoint

We want to monitor every API endpoint our application exposes. Remember that different HTTP methods define different API endpoints. For example:

GET /user/:id
PUT /user/:id

The above count as two separate endpoints, even though the URL is the same.

Cover key API parameters

Some parameters can change the endpoint's response significantly. We shoud strive to have separate checks verifying that the endpoint is behaving correctly across different configurations.

Keep checks focused & independent

API monitoring checks must be organised as to minimise the time needed to identify resolve the underlying issue. This means we need to keep our checks focused on a specific case (vs trying to have a single check do many things) and independent from each other (vs building chains of checks that build on top of one another).

Scheduled global API checks

Checkly specialises in API monitoring and allows users to run API checks on a schedule from global locations. We can combine these checks with custom alerting to be able to quickly respond and remediate potential API issues.

A Checkly API check is comprised of the following components.

Main HTTP request

The most basic building block of Checkly's API check is the main HTTP request. This can be fully configured in its method, url, parameters and body to fully reproduce a real-world web API call.

Assertions

Assertions allow us to check for every key aspect of the response. A check with one or more failing assertions will enter failing state and trigger any connected alert channel.

In this example, we are checking against:

The status code, expected to be 200.
The id of one of the customers returned as part of the response's JSON body. Here we could assert a specific value, but in this case we are happy with just verifying that the field is not empty.
The value of the Content-Encoding header, expected to equal gzip.

Response time limits

Response time limits enable us to set different thresholds to decide exactly which response time maps to a hard failure, a pass or a degradation. We can use transitions between these states to trigger different kinds of alerts using our preferred channels.

Setup and teardown scripts

Checkly is highly programmable and allows users to run scripts before and after the main HTTP request of an API check.

Setup scripts run before our check and give us access to properties like the URL, headers and query parameters, enabling us to set up all the prerequisites for a successful request. Some examples could be:

Fetching a token from a different API endpoint.
Setting up test data on the target system.
Formatting data to be sent as part of the request.

Teardown scripts run after the request has executed, bur right before the assertions. They are useful for manipulating the response (for example to remove sensitive information) or removing any test data on the target system.

Improving our monitoring

As we increase our monitoring coverage across our APIs, we can also increase the efficacy of our setup by:

Importing existing Swagger/OpenAPI specs or even cURL commands using built-in functionality.
Defining our API checks as code to scale our setup while lowering maintenance needs.
Combining our API checks with E2E monitoring for any website or web app service whose API we might be monitoring.

Banner image: “rover 200 framing line” by spencer_cooper is licensed under CC BY-ND 2.0

End to end monitoring

Tim Nolet 👨🏻‍🚀 — Mon, 29 Mar 2021 18:20:13 +0000

This article originally appeared in Checkly's Guides.

End-to-end monitoring uses headless browser automation tools like Puppeteer and Playwright to continuously test your website's key user flows. This article summarises the most important points on this topic and gets you up and running in 10 minutes.

Headless browser testing

Over the course of the last decade, especially thanks to tools such as Selenium and (more recently) Cypress, automated End-to-End testing (E2E testing) has become widespread across industries.

Broadly speaking, E2E testing entails running fully automated test suites with the goal of catching bugs before they hit production and, therefore, negatively affect the user experience. These test suites need to be carefully scripted using dedicated tools, as well as to be made stable and fast enough to test the most important end-user flows on every build, PR or commit, depending on the application under test and the organisation's automation maturity.

The industry has learned to struggle with the challenges this approach presents:

Long-running suites.
Test flakiness.
Expensive test infrastructure.

All of the above lead to higher costs and slower delivery.

The appearance of mature headless browser automation tools, such as Puppeteer and Playwright, offers a response to many of the above issues by allowing testing in the browser without its GUI, which yields higher speed and stability coupled with lower resource consumption.

E2E monitoring (AKA synthetic monitoring)

While this nimbler, more reliable kind of test is already a big improvement for pre-production testing, it enables a completely new approach in production monitoring: we can now continuously run E2E tests against our production systems. This enables us to have real-time feedback on the status of our website's key user flows from a user's perspective. This is E2E monitoring, also known as synthetic monitoring or active monitoring.

This comes with a significant, often underestimated advantage: it allows us to catch all those things that might break in production that can't be caught during pre-production testing. We are now running directly against the system that the end-user is actually interacting with, and will be able to monitor its behaviour in real time.

What could this look like in practice? Let's look at an e-commerce example.

Monitoring a web shop

A few key flows for an e-commerce websites could be:

Logging in
Finding a product through search
Adding products to the basket and checking out

Let's see how to set them up - for this example, we will do that on our demo web shop.

Playwright E2E tests

Using Playwright, we can script our three E2E scenarios as follows.

Login scenario:

const { chromium } = require("playwright");

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // click on the login button and go through the login procedure
  await page.click("#login");
  await page.type("#n-email", "user@email.com");
  await page.type("#n-password2", "supersecure1");
  await page.click("#goto-signin-btn");

  // wait until the login confirmation message is shown
  await page.waitForSelector("#login-message", { visible: true });

  // close the browser and terminate the session
  await browser.close();
})();

Search scenario:

const { chromium } = require("playwright");
const assert = require("chai").assert;

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  const bookList = [
    "The Foreigner",
    "The Transformation",
    "For Whom the Ball Tells",
    "Baiting for Robot",
  ];

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // search for keyword
  await page.click(".topbar > input");
  await page.type(".topbar > input", "for");
  await page.click("#button-search");
  await page.waitForSelector(
    ".shop-content > ul > .preview:nth-child(1) > .preview-title"
  );

  // halt immediately if results do not equal expected number
  let resultsNumber = (await page.$$(".preview-title")).length;
  assert.equal(resultsNumber, bookList.length);

  // remove every element found from the original array...
  for (i = 0; i < resultsNumber; i++) {
    const resultTitle = await page.$eval(
      `.preview:nth-child(${i + 1}) > .preview-title`,
      (e) => e.innerText
    );

    const index = bookList.indexOf(resultTitle);
    bookList.splice(index, 1);
  }

  // ...then assert that the original array is now empty
  assert.equal(bookList.length, 0);

  // close the browser and terminate the session
  await browser.close();
})();

Checkout scenario:

const { chromium } = require("playwright");

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  const navigationPromise = page.waitForNavigation();

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // add the first item to the cart
  await page.click(`.preview:nth-child(1) > .preview-author`);
  await page.click(".detail-wrapper > .call-to-action");
  await page.click("#logo");

  // wait until navigation is complete
  await navigationPromise;

  // navigate to cart and proceed
  await page.click("#cart");
  await page.click(".cart > .call-to-action");
  await page.click("#s-name");

  // fill out checkout info
  await page.type("#s-name", "Max");
  await page.type("#s-surname", "Mustermann");
  await page.type("#s-address", "Charlottenstr. 57");
  await page.type("#s-zipcode", "10117");
  await page.type("#s-city", "Berlin");
  await page.type("#s-company", "Firma GmbH");
  await page.click(".checkout > form");
  await page.click("#asap");

  // confirm checkout
  await page.click(".checkout > .call-to-action");

  // wait until the order confirmation message is shown
  await page.waitForSelector("#order-confirmation", { visible: true });

  // close the browser and terminate the session
  await browser.close();
})();

These can be run on our own machine without issues with very little preparation with a simple node script.js.

Monitoring application performance

A web application's performance plays a primary role in the user experience it delivers. From the user's perspective, a fully functional application that is not performant quickly becomes indistinguishable from a broken one.

Using Playwright together with browser APIs or additional performance libraries, our end-to-end monitoring setup can be easily extended to include application performance.

Measuring execution time

An effective and granular way to gauge performance is to measure how long our scenario takes to execute. A very simple way to achieve this is to just time our script's execution with time node script.js.

Oftentimes it pays to be more granular. For example, we might want to measure the durations of certain segments of a given flow and assert against them. We can do all this in our script. For example, in the case of our longer checkout example:

const { chromium } = require("playwright");
// we add an assertion library
const assert = require("chai").assert; 

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  const navigationPromise = page.waitForNavigation();

  // get first timestamp
  const tFirstNavigationStarts = Date.now()

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // get second timestamp
  const tFirstNavigationEnds = Date.now()

  // add the first item to the cart
  await page.click('.preview:nth-child(1) > .preview-author');
  await page.click(".detail-wrapper > .call-to-action");
  await page.click("#logo");

  // ...

  // wait until the order confirmation message is shown
  await page.waitForSelector("#order-confirmation", { visible: true });

  // get thirds timestamp
  const tScenarioEnds = Date.now();

  // calculate timings
  const dNavigation = tFirstNavigationEnds - tFirstNavigationStarts;
  const dScenario = tScenarioEnds - tFirstNavigationStarts ;

  // assert against the timings we have measured
  assert.isBelow(dNavigation, 1750, 'Initial navigation took longer than 1.75s')
  assert.isBelow(dScenario, 3000, 'Total scenario took longer than 3s')

  // close the browser and terminate the session
  await browser.close();
})();

We can also use Web Performance APIs such as Navigation Timing and Resource Timing, as well as libraries such as Google Lighthouse. For more examples, see our dedicated performance guide.

End to end application monitoring

Unlike headful tools, headless ones tend to not be very resource-hungry, which makes it easier to move our scripts to the cloud. Checkly runs on top of AWS Lambda, and enables us to quickly copy-paste our script and set it up to run on a schedule from locations around the world.

We can move our scripts to separate checks to keep them independent - we want to optimise for parallelisation and clarity of feedback.

As soon as a check runs red, we are alerted in real time and can intervene before the issue impacts our users. Alerting can be set up with all the industry-standard channels like Pagerduty, Opsgenie, Slack, email, SMS and more.

On-demand checking

Active monitoring and event-triggered testing do not exclude one another. You might want to have checks kicked off every time you deploy to production, or on every merge, PR or commit, or you might also want to run against your staging or development server. The choice has to be made based on your workflow and on your automation strategy.

CI/CD

Tests can be kicked off of CI pipelines. You might want to use different hooks (for e.g. smoke vs regression testing) in different stages and against different targets. Checkly supports all major CI servers.

Develop-preview-test

If you are using provider like Vercel, you can automatically trigger your checks to run on deployed PRs, too, to reap the benefits of the develop-preview-test approach.

Pitfalls

We learned things the hard way so you do not have to. When starting out, keep an eye out for the following pitfalls:

Non-independent tests: tests that rely on one another in any way (e.g. order of execution, test data) are hard to parallelise, resulting in longer execution times and potentially higher flakiness. Keep your tests independent.
Long, unfocused tests: checking too much in a single test will make failures harder to debug. Break it up instead, and enjoy the added parallelisation.
Messing up your own metrics, KPIs: remember, if you are not running against production, you want to make sure your E2E monitoring checks or tests are filtered out of your analytics. This is rather easy to do, with most headless browser tools normally identifying themselves as such from the start.

Banner image: detail from "Outdoor Gas Installation" by christian.senger is licensed under CC BY-SA 2.0

DEV Community: Checkly

The Advent of Monitoring, Day 3: Easy Monitoring for Self-Hosted Projects with Checkly

Setting Up Multi-Step API Checks

The Advent of Monitoring, Day 2: Debugging Dashboard Outages with Checkly's API Checks

Our Hypothesis

ChatGPT to the Rescue

Configuring API Checks with Checkly

The Advent of Monitoring, Day 1: What Are Synthetics and Why They Are Needed

How does all of this relate to synthetics?

Using Playwright to Monitor Third-Party Resources That Could Impact User Experience

Examples of problematic external resource usage

Slow and render-blocking resources

Tightly coupled JavaScript dependencies

How to test and monitor risky third-party resources with Playwright

How to emulate slow third-party resources with Playwright

How to block requests with Playwright

So what’s next?

Frontend testing vs. monitoring

Conclusion

Playwright Tips From the Checkly Community

How should you handle parallelism in your end-to-end tests?

What are common reasons for flaky tests and how can you avoid them?

How can you emulate custom keyboard interactions in Playwright?

How can you catch JavaScript errors in long end-to-end testing flows?

Conclusion

How low-level API calls can stabilize your end-to-end tests

Plain API calls in end-to-end browser tests

A custom Node.js API wrapper

How to extract resource ids using Playwright request interception

Wrapping Up

Avoiding hard waits in Playwright and Puppeteer

The problems with hard waits

How to fix it

Built-in waits

Explicit waits

Waiting on navigations and network conditions

Waiting for an element

Waiting on page events

Waiting on page functions

Takeaways

Migrating from Puppeteer to Playwright

Puppeteer and Playwright today

Why switch

What to change in your scripts - short version

What to change in your scripts - in depth

Require Playwright package

The browser context

Waiting

Setting viewport

Typing

Cookies

XPath selectors

Device emulation

File download

File upload

Request interception

New possibilities to be aware of

New selector engines

Saving and reusing state

Locator API

Playwright Inspector

Playwright Test

Trace Viewer

Test Generator

Switching to Playwright for richer browser check results

Debugging Puppeteer & Playwright scripts

Awareness comes first

The error message

Gaining visibility

Increasing logging

Accessing DevTools

The Playwright Inspector

Further reading

OpenAPI/Swagger Monitoring

The OpenAPI Specification

Swagger vs OpenAPI

Exploring the specification

Generating documentation from OpenAPI

Generating boilerplate code from OpenAPI

Generated monitoring checks with Checkly