DEV Community: Tim Nolet 👨🏻‍🚀

Avoiding hard waits in Playwright and Puppeteer

Tim Nolet 👨🏻‍🚀 — Thu, 16 Dec 2021 15:39:13 +0000

Looking to solve the issue of a page or element not being loaded, many take the shortcut of waiting for a fixed amount of time - adding a hard wait, in other words. This is regarded as an anti-pattern, as it lowers performance and increases the chances of a script breaking (possibly intermittently). Let's explore how those issues arise and what better solutions we can use to avoid them.

The problems with hard waits

Hard waits do one thing and one thing only: wait for the specified amount of time. There is nothing more to them. This makes them dangerous: they are intuitive enough to be favoured by beginners and inflexible enough to create serious issues.

Let's explore these issues in practical terms through an example. Imagine the following situation: our script is running using a tool without any sort of built-in smart waiting, and we need to wait until an element appears on a page and then attempt to click it. We try to solve this issue with a hard wait, like Puppeteer's page.waitFor(timeout).

This could looks something like the following:

await page.waitFor(1000); // hard wait for 1000ms
await page.click('#button-login');

In such a situation, the following can happen:

1) We can end up waiting for a shorter amount of time than the element takes to load!

In this case, our hard wait terminates and our click action is attempted too early. The script terminates with an error, possibly of the "Element not found" sort.

2) The element can load before our hard wait has expired.

While the element is correctly clicked once our wait expires, and our script continues executing as planned, we are wasting precious time - likely on each hard wait we perform. Across multiple scripts and suites, this can add up to noticeable drag on build time.

In general, with hard waits we are virtually always waiting too little or too long. In the worst case scenario, the fluctuations in load time between different script executions are enough to make the wait sometimes too long and sometimes too short (meaning we will switch between scenario 1 and 2 from above in an unpredictable manner), making our script fail intermittently. That will result in unpredictable, seemingly random failures, also known as flakiness.

Flakiness, a higher-than-acceptable false failure rate, can be a major problem. It is essentially a source of noise, making it harder to understand what the state of the system we are testing or monitoring really is. Not only that, but stakeholders who routinely need to investigate failures only to find out that they are script-related (instead of system-related) will rapidly lose confidence in an automation setup.

How to fix it

To avoid these issues, we have to ditch hard waits completely outside debugging scenarios. That means that hard waits should never appear in production scripts under any circumstance.

Our aim should be to wait just long enough for the element to appear. We want to always be certain the element is available, and never waste any time doing that. Luckily most automation tools and frameworks today offer multiple ways to achieve this. We can call these "smart waits".

Different tools approach the broad topic of waiting in different ways. Both Puppeteer and Playwright offer many different kinds of smart waits, but Playwright takes things one step further and introduces an auto-waiting mechanism on most page interactions.

Let's take a look at different smart waiting techniques and how they are used.

Built-in waits

Playwright comes with built-in waiting mechanisms on navigation and page interactions. Since these are baked into the tool itself, it is good to get familiar with the logic behind them, as well as how to override the default behaviour when necessary.

Explicit waits

Explicit waits are a type of smart wait we invoke explicitly as part of our script. We will want to use them more or less often depending on whether our automation tool has a built-in waiting mechanism (e.g. Playwright) or requires us to handle all the waiting (e.g. Puppeteer).

If you can rely on automatic waits, use explicit waits only when necessary. An auto-wait system failing once is no good reason for ditching the approach completely and adding explicit waits before every page load and element interaction. If the tool you are using does not do auto-waiting, you will be using explicit waits quite heavily (possibly after each navigation and before each element interaction), and that is fine - there is just less work being done behind the scenes, and you are therefore expected to take more control into your hands.

Waiting on navigations and network conditions

On a page load, we can use the following:

page.waitForNavigation to wait until a page navigation (new URL or page reload) has completed.
page.waitForLoadState for Playwright, waits until the required load state has been reached (defaults to load); page.waitForNetworkIdle with Puppeteer, a narrower method to wait until all network calls have ended.
page.waitForURL with Playwright, waits until a navigation to the target URL.

All the above default to waiting for the load event, but can also be set to wait for:

the DOMContentLoaded event.
Playwright only: networkidle, raised when there are no network connections for at least 500 ms.
Playwright only: commit, when the network response is received and the document starts loading (Playwright only).
Puppeteer only: networkidle0, raised when there are no network connections for at least 500 ms.
Puppeteer only: networkidle2, raise when the there are no more than 2 network connections for at least 500 ms.

Lazy-loaded pages might require extra attention when waiting for the content to load, often demanding explicitly waiting for specific UI elements. See the following section.

Additionally, we can also wait until a specific request is sent out or a specific response is received with page.waitForRequest and page.waitForResponse. These two methods are key for implementing request and response interception.

Waiting for an element

We can also explicitly wait for a specific element to appear on the page. This is normally done via page.waitForSelector or a similar method, like page.waitForXPath (Puppeteer only). A good knowledge of selectors is key to enable us to select precisely the element we need to wait for.

Waiting on page events

With Playwright, we can also directly wait on page events using page.waitForEvent.

Waiting on page functions

For more advanced cases, we can pass a function to be evaluated within the browser context via page.waitForFunction.

Takeaways

Never use hard waits outside of debugging
Use smart waits instead, choosing the best one for your situation
Use more or less smart waits depending on whether your tool support auto-waits

Banner image: detail from "IMG_0952" by sean_emmett is licensed under CC BY-NC-SA 2.0

Migrating from Puppeteer to Playwright

Tim Nolet 👨🏻‍🚀 — Tue, 16 Nov 2021 17:13:26 +0000

This article originally appeared in Checkly's Guides.

Puppeteer and Playwright today

While they share a number of similarities, Puppeteer and Playwright have evolved at different speeds over the last two years, with Playwright gaining a lot of momentum and arguably even leaving Puppeteer behind.

These developments have led many to switch from Puppeteer to Playwright. This guide aims to show what practical steps are necessary and what new possibilities this transition enables. Do not let the length of this article discourage you - in most cases the migration is quick and painless.

Why switch

While a comprehensive comparison of each tool's strengths and weaknesses could fill up a guide of its own (see our previous benchmarks for an example), the case for migrating to Playwright today is rather straightforward:

As of the writing of this guide, Playwright has been frequently and consistently adding game changing features (see below for a partial list) for many months, with Puppeteer releasing in turn mostly smaller changes and bug fixes. This led to a reversal of the feature gap that had once separated the two tools.
Playwright maintains an edge in performance in real-world E2E scenarios (see benchmark linked above), resulting in lower execution times for test suites and faster monitoring checks.
Playwright scripts seem to run even more stable than their already reliable Puppeteer counterparts.
The Playwright community on GitHub, Twitter, Slack and beyond has gotten very vibrant, while Puppeteer's has gone more and more quiet.

What to change in your scripts - short version

Below you can find a cheat sheet with Puppeteer commands and the corresponding evolution in Playwright. Keep reading for a longer, more in-depth explanation of each change.

Remember to add await as necessary.

Puppeteer	Playwright
`require('puppeteer')`	`require('playwright')`
`puppeteer.launch(...)`	`playwright.chromium.launch(...)`
`browser.createIncognitoBrowserContext(...)`	`browser.newContext(...)`
`page.setViewport(...)`	`page.setViewportSize(...)`
`page.waitForSelector(selector)` `page.click(selector);`	`page.click(selector)`
`page.waitForXPath(XPathSelector)`	`page.waitForSelector(XPathSelector)`
`page.$x(xpath_selector)`	`page.$(xpath_selector)`
`page.waitForNetworkIdle(...)`	`page.waitForLoadState({ state: 'networkidle' }})`
`page.waitForFileChooser(...)`	Removed, handled differently.
`page.waitFor(timeout)`	`page.waitForTimeout(timeout)`
`page.type(selector, text)`	`page.fill(selector, text)`
`page.cookies([...urls])`	`browserContext.cookies([urls])`
`page.deleteCookie(...cookies)`	`browserContext.clearCookies()`
`page.setCookie(...cookies)`	`browserContext.addCookies(cookies)`
`page.on('request', ...)`	Handled through page.route.
`elementHandle.uploadFile(...)`	`elementHandle.setInputFiles(...)`
Tricky file download.	Better support for downloads.

Did we forget anything? Please let us know by getting in touch, or submit your own PR.

What to change in your scripts - in depth

Require Playwright package

In Puppeteer, the first few lines of your script would have most likely looked close to the following:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // ...

With Playwright you do not need to change much:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // ...

Playwright offers cross-browser support out of the box, and you can choose which browser to run with just by changing the first line, e.g. to const { webkit } = require('playwright');
In Puppeteer, this would have been done throught the browser's launch options:

  const browser = await puppeteer.launch({ product: 'firefox' })

The browser context

Browser contexts already existed in Puppeteer:

const browser = await puppeteer.launch();
const context = await browser.createIncognitoBrowserContext();
const page = await context.newPage();

Playwright's API puts even more importance on them, and handles them a little differently:

const browser = await chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();

Like in Puppeteer, for basic cases and single-page flows, you can use the default context:

const browser = await chromium.launch();
const page = await browser.newPage();

When in doubt, explicitly create a new context at the beginning of your script.

Waiting

The auto-waiting mechanism in Playwright means you will likely not need to care about explicitly waiting as often. Still, waiting being one of the trickiest bits of UI automation, you will still want to know different ways of having your script explicitly wait for one or more conditions to be met.

In this area, Playwright brings about several changes you want to be mindful of:

page.waitForNavigation and page.waitForSelector remain, but in many cases will not be necessary due to auto-waiting.
page.waitForEvent has been added.
Puppeteer's page.waitForXPath has been incorporated into page.waitForSelector, which recognises XPath expressions automatically.
page.waitForFileChooser been removed removed (see the official dedicated page and our file upload example for new usage)
page.waitForNetworkIdle has been generalised into page.waitForLoadState (see the networkidle state to recreate previous behaviour)
page.waitForUrl has been added allowing you to wait until a URL has been loaded by the page's main frame.
page.waitFor(timeout) becomes page.waitForTimeout(timeout).

This is as good a place as any to remind that page.waitForTimeout should never be used in production scripts! Hard waits/sleeps should be used only for debugging purposes.

Setting viewport

Puppeteer's page.setViewport becomes page.setViewportSize in Playwright.

Typing

While puppeteer's page.type is available in Playwright and still handles fine-grained keyboard events, Playwright adds page.fill specifically for filling and clearing forms.

Cookies

With Puppeteer cookies are handled at the page level; with Playwright you manipulate them at the BrowserContext level.

The old...

...become:

Note the slight differences in the methods and how the cookies are passed to them.

XPath selectors

XPath selectors starting with // or .. are automatically recognised by Playwright, whereas Puppeteer had dedicated methods for them. That means you can use e.g. page.$(xpath_selector) instead of page.$x(xpath_selector), and page.waitForSelector(xpath_selector) instead of page.waitForXPath(xpath_selector). The same holds true for page.click and page.fill.

Device emulation

Playwright device emulation settings are set at Browser Context level, e.g.:

const pixel2 = devices['Pixel 2'];
const context = await browser.newContext({
  ...pixel2,
});

On top of that, permission, geolocation and other device parameters are also available for you to control.

File download

Trying to download files in Puppeteer in headless mode can be tricky. Playwright makes this more streamlined:

const [download] = await Promise.all([
  page.waitForEvent('download'),
  page.click('#orders > ul > li:nth-child(1) > a')
])

const path = await download.path();

See our example on file download.

File upload

Puppeteer's elementHandle.uploadFile becomes elementHandle.setInputFiles.

See our example on file upload.

Request interception

Request interception in Puppeteer is handled via page.on('request', ...):

await page.setRequestInterception(true)

page.on('request', (request) => {
  if (request.resourceType() === 'image') request.abort()
  else request.continue()
})

In Playwright, page.route can be used to intercept requests with a URL matching a specific pattern:

await page.route('**/*', (route) => {
  return route.request().resourceType() === 'image'
    ? route.abort()
    : route.continue()
})

See our full guide on request interception for more examples.

For many of the points in the list above, variations of the same function exist at Page, Frame and ElementHandle level. For simplicity, we reported only one.

New possibilities to be aware of

When moving from Puppeteer to Playwright, make sure you inform yourself about the many completely new features Playwright introduces, as they might open up new solutions and possibilities for your testing or monitoring setup.

New selector engines

Playwright brings with it added flexibility when referencing UI elements via selectors by exposing different selector engines. Aside from CSS and XPath, it adds:

Playwright-specific selectors, e.g.: :nth-match(:text("Buy"), 3)
Text selectors, e.g.: text=Add to Cart
Chained selectors, e.g.: css=preview >> text=In stock

You can even create your own custom selector engine.

For more information on selectors and how to use them, see our dedicated guide.

Saving and reusing state

Playwright makes it easy for you to save the authenticated state (cookies and localStorage) of a given session and reuse it for subsequent script runs.

Reusing state can save significant amounts of time on larger suites by skipping the pre-authentication phase in scripts where it is not supposed to be directly tested / monitored.

Locator API

You might be interested in checking out Playwright's Locator API, which encapsulates the logic necessary to retrieve a given element, allowing you to easily retrieve an up-to-date DOM element at different points in time in your script.

This is particularly helpful if you are structuring your setup according to the Page Object Model, or if you are interested to do start doing that.

Playwright Inspector

The Playwright Inspector is a GUI tool that comes in very handy when debugging scripts, allowing you to step instruction-by-instruction through your script to more esily identify the cause of a failure.

The Inspector also comes in handy due its ability to suggest selectors for page elements and even record new scripts from scratch.

Playwright Test

Playwright comes with its own runner, Playwright Test, which adds useful features around end-to-end testing, like out-of-the-box parallelisation, test fixtures, hooks and more.

Trace Viewer

The Playwright Trace Viewer allows you to explore traces recorded using Playwright Test or the BrowserContext Tracing API. Traces are where you can get the most fine-grained insights into your script's execution.

Test Generator

You can use the Playwright Test Generator to record interactions in your browser. The output will be a full-fledged script ready to review and execute.

Switching to Playwright for richer browser check results

Checkly users switching to Playwright can take advantage of its new Rich Browser Check Results, which come with tracing and Web Vitals and make it easier to isolate the root cause of a failed check and remediate faster.

This reveals additional information about the check execution, including:

Overview of all errors raised (console, network and script errors)
A timeline summarising the execution across page navigations
For each page visited, a network & timing timeline, Web Vitals, console and network tabs.
In case of a failing check, a screenshot on failure.

Aside from running a Playwright script, performance and error tracing also require the use of Runtime 2021.06 or newer.

Note that cross-browser support is not available on Checkly - our Browser checks run on Chromium only.

banner image: "rocket engine" by industrial arts, licensed under CC BY 2.0

Debugging Puppeteer & Playwright scripts

Tim Nolet 👨🏻‍🚀 — Tue, 10 Aug 2021 14:27:03 +0000

This article originally appeared on Checkly's Learn Puppeteer & Playwright

Understanding why a script does not work as expected, or just what the root cause of a failure is, is a key skill for automation. Given its importance and its sometimes deceptive complexity, debugging is a topic that should receive quite some attention.

This article will explore basic concepts and tools to point beginners in the right direction.

Awareness comes first

Script debugging is firstly about observing and understanding. Finding out what is causing the failure (or misbehaviour) in your execution heavily depends on your knowledge of:

What the script you are looking at is supposed to do
How the application the script is running against is supposed to behave at each step of the script

When approaching a debugging session, make sure the above points are taken care of. Skipping this step is way more likely to cost you additional time than it is to save you any.

The error message

Error messages are not present in every scenario: we might be trying to understand why a script passes, or why it takes longer than expected. But when we have access to an error message, we can use it to guide us.

The error, in and of its own, is not always enough to understand what is going wrong with your script. Oftentimes, there can be multiple degrees of separation between the error and its root cause. For example: an "Element not found" error might be alerting you to the fact that an element is not being found on the page, but that itself might be because the browser was made to load the wrong URL in the first place.

Do not fall into the easy trap of reading the error message and immediately jumping to conclusions. Rather, take the error message, research it if needed, combine it with your knowledge of script and app under test and treat it as the first piece to the puzzle, rather than the point of arrival of your investigation.

Good knowledge of the automation tool you are using will also help add more context to the error message itself.

Gaining visibility

Given that Headless browser scripts will run without a GUI, a visual assessment of the state of the application needs additional steps.

One possibility is to adding screenshots in specific parts of the script, to validate our assumptions on what might be happening at a given moment of the execution. For example, before and after a problematic click or page transition:

For Playwright:

...
await page.screenshot({ path: 'before_click.png' });
await page.click('#button')
await page.screenshot({ path: 'after_click.png' });
...

For Puppeteer:

...
await page.screenshot({ path: 'before_click.png' });
await page.waitForSelector('#button');
await page.click('#button')
await page.screenshot({ path: 'after_click.png' });
...

Another way to better observe our script's execution is to run in headful mode:

For Playwright:

...
const browser = await chromium.launch({ headless: false, slowMo: 20 });
...

For Puppeteer:

...
const browser = await puppeteer.launch({ headless: false, slowMo: 20 });
...

We can then tweak the slowMo option, which adds a delay in milliseconds between interactions, to make sure the execution is not too fast for us to follow.

Increasing logging

Sometimes we need to try and see the execution through our automation tool's eyes. Added logging can help by taking us step-by-step through every command as it is executed.

For Playwright:

DEBUG=pw:api node script.js

For Puppeteer:

DEBUG="puppeteer:*" node script.js

Accessing DevTools

A wealth of information is available through the Chrome Developer Tools. We can configure our browser to start with the DevTools tab already open (this will automatically disable headless mode), which can be helpful when something is not working as expected. Careful inspection of the Console, Network and other tabs can reveal hidden errors and other important findings.

For Playwright:

...
await chromium.launch({ devtools: true });
...

For Puppeteer:

...
await browser.launch({ devtools: true });
...

We can also use the console to directly try out a selector on the page in its current state, e.g. with document.querySelector or document.querySelectorAll.

If we are using Playwright, we can also run in debug mode with PWDEBUG=console node script.js. This provisions a playwright object in the browser which allows us to also try out Playwright-specific selectors.

The Playwright Inspector

The Playwright Inspector is a GUI tool which exposes additional debugging functionality, and can be launched using PWDEBUG=1 npm run test.

The Inspector allows us to easily step through each instruction of our script, while giving us clear information on the duration, outcome, and functioning of each. This can be helpful in getting to the root cause of some of the more generic errors.

The Inspector includes additional handy features such as selector generation and debugging, as well as script recording.

OpenAPI/Swagger Monitoring

Tim Nolet 👨🏻‍🚀 — Wed, 28 Apr 2021 14:04:24 +0000

OpenAPI and Swagger help users design and document APIs in a way that is readable from both humans and machines. As a consequence, they can also be used to generate the code that will run the specified API - both on the provider and consumer side. Can we leverage this same principle to simplify API monitoring? After a brief first look at OpenAPI and Swagger, this article will show how we can quickly use them to monitor a new or existing API.

The OpenAPI Specification

The OpenAPI Specification (OAS) specifies a standard, language-agnostic and machine-readable format to describe a web API in one or more files. Nowadays, it acts as a vendor-neutral standard for describing the structure and behaviour of HTTP-based APIs, and exists as part of the OpenAPI Initiative (OAI).

Following the OAS brings the following advantages:

A description of the API that is readable from both humans and machines.
The possibility of automatically generating documentation, implementation logic and even mock servers for testing.
Early validation of the data flows happening within the API.

For those new to the OAS and wanting to get a basic understanding without diving straight into the specification itself, the official OpenAPI Specification Explained guide is a great place to start.

Swagger vs OpenAPI

The OpenAPI Specification is based on the SmartBear SoftwareSmartBear Software, which had previously been a project driven by SmartBear Software. SmartBear donated the Swagger Specification to the Linux Foundation in 2015. In a sense, the OAS is the newer, fully open source-driven incarnation of the Swagger Specification.

The name Swagger today indicates a set of tools, both free and paid, that support users of the OpenAPI ecosystem.

Exploring the specification

The core of our API specification will happen in the OpenAPI root document, normally named openapi.json or openapi.yml. We can choose between JSON and YAML formats as we like.

The bare minimum we need to declare is the OpenAPI version, the basic information about our API, and finally the available endpoints.

openapi: 3.0.0
info:
  title: OpenAPI document for our new Payment API
  version: 0.0.1
paths: {} # Nothing here yet

An API without endpoints is not very useful. We can add our first, together with a GET operation:

openapi: 3.0.0
info:
  title: OpenAPI document for our new Payment API
  version: 0.0.2
paths:
  # User account balance
  /balance/{id}:
    # Retrieve the user's balance on the given account
    get: 
      ...

The next step is to define what the operation will look like. In this case, we will specify some constraints on the id parameter, as well as the response.

openapi: 3.0.0
info:
  title: OpenAPI document for our new Payment API
  version: 0.0.3
paths:
  # User account balance
  /balance/{id}:
    # Retrieve the user's balance on the given account
    get: 
      summary: Get the user's account balance
      description: Retrieves the current account balance for the given user.
      operationId: get-balance
      parameters:
      - name: id
        in: path
        required: true
        schema:
          type: string
          format: uuid
      responses:
        "200":
          description: "OK"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/balance"

Notice we are explicitly defining which schemas the parameter and response should conform to. In the case of our response, we are referring to an
externally defined schema. This is helpful when we need to reuse a schema definition multiple times across our spec file.

As we proceed, we might add multiple endpoints, each with one or more operations.

openapi: 3.0.0
info:
  title: OpenAPI document for our new Payment API
  version: 0.0.4
paths:
  # User account balance
  /balance/{id}:
    # Retrieve the user's balance on the given account
    get:
      summary: Get the user's account balance
      ...
  /transactions:
    get:
      summary: Retrieve transaction
      ...
    post:
      summary: Submit transaction
      ...
  ...

We can keep on building out our API spec as needed. The more precisely we describe our API in this file, the more useful this specification will become, whether we use it for documentation or code generation purposes.

Generating documentation from OpenAPI

Let's now take a look at a more complex, finished example: the Swagger PetStore demo. Even if we had written it ourselves, this description of a complete API could be overwhelming if we just tried to parse the file line by line. To first-time users, that could feel even more daunting. A better option would be to use the open-source toolset Swagger offers to generate human-readable, interactive API documentation from the file we already have.

Pasting the content of the spec file to Swagger Editor will produce a preview of our Swagger UI documentation. This is helpful for consumers of the API, who will be presented with an orderly documentation page breaking down each endpoint while also allowing users to test out different operations.

Generating boilerplate code from OpenAPI

When writing a new API, we will oftentimes need to produce a certain amount of boilerplate code for both our provider (e.g. a server exposing our API for consumption) and consumer (e.g. an SDK for our API) applications. Once we have our OpenAPI file in place, we can use Swagger Codegen (also available within Swagger Editor) to automatically generate that code for us, saving us precious time and reducing the avenues for human error.

The following snippet is an example of the code Swagger Codegen can generate starting from an OAS file.

/**
   * Invokes the REST service using the supplied settings and parameters.
   * @param {String} path The base URL to invoke.
   * @param {String} httpMethod The HTTP method to use.
   * @param {Object.<String, String>} pathParams A map of path parameters and their values.
   * @param {Object.<String, Object>} queryParams A map of query parameters and their values.
   * @param {Object.<String, Object>} collectionQueryParams A map of collection query parameters and their values.
   * @param {Object.<String, Object>} headerParams A map of header parameters and their values.
   * @param {Object.<String, Object>} formParams A map of form parameters and their values.
   * @param {Object} bodyParam The value to pass as the request body.
   * @param {Array.<String>} authNames An array of authentication type names.
   * @param {Array.<String>} contentTypes An array of request MIME types.
   * @param {Array.<String>} accepts An array of acceptable response MIME types.
   * @param {(String|Array|ObjectFunction)} returnType The required type to return; can be a string for simple types or the
   * constructor for a complex type.
   * @param {module:ApiClient~callApiCallback} callback The callback function.
   * @returns {Object} The SuperAgent request object.
   */
  exports.prototype.callApi = function callApi(path, httpMethod, pathParams,
      queryParams, collectionQueryParams, headerParams, formParams, bodyParam, authNames, contentTypes, accepts,
      returnType, callback) {

    var _this = this;
    var url = this.buildUrl(path, pathParams);
    var request = superagent(httpMethod, url);

    // apply authentications
    this.applyAuthToRequest(request, authNames);

    // set collection query parameters
    for (var key in collectionQueryParams) {
      if (collectionQueryParams.hasOwnProperty(key)) {
        var param = collectionQueryParams[key];
        if (param.collectionFormat === 'csv') {
          // SuperAgent normally percent-encodes all reserved characters in a query parameter. However,
          // commas are used as delimiters for the 'csv' collectionFormat so they must not be encoded. We
          // must therefore construct and encode 'csv' collection query parameters manually.
          if (param.value != null) {
            var value = param.value.map(this.paramToString).map(encodeURIComponent).join(',');
            request.query(encodeURIComponent(key) + "=" + value);
          }
        } else {
          // All other collection query parameters should be treated as ordinary query parameters.
          queryParams[key] = this.buildCollectionParam(param.value, param.collectionFormat);
        }
      }
    }

    // set query parameters
    if (httpMethod.toUpperCase() === 'GET' && this.cache === false) {
        queryParams['_'] = new Date().getTime();
    }
    request.query(this.normalizeParams(queryParams));

  ...

Depending on how much is described in a given API spec, it is possible to move even further and generate tests, mock servers and more. See openapi.tools for an up-to-date list of tools belonging to the OpenAPI ecosystem.

Generated monitoring checks with Checkly

API monitoring is key to ensure that our endpoints are returning the correct results in an acceptable timeframe. If we can generate APIs from a OAS description file, we can also generate monitoring checks for them. Checkly is a member of the OpenAPI initiative, and allows us to import an OpenAPI spec file and automatically creates one or more checks depending on the amount of information available. All it takes is a couple of clicks.

When creating an API check, select the import from Swagger / OpenAPI button.

That will bring up the import dialog. This is where we give Checkly the URL to the Swagger/OpenAPI specification file.

Before anything is created on our behalf, Checkly will show us a summary of all the checks that will be imported. We can choose to import them all defined checks or just a subset.

Notice that we can also select whether to import headers, query params and other settings or just to ignore them for now.

Once we confirm, the checks will be created for us and will start running according to the schedule we have selected.

Based on how comprehensively the file we imported described the API we want to monitor, we might have to tweak certain checks manually before they are fully set up.

This usually includes adding missing headers or body contents that might not have been explicitly defined in the imported file.

We can open the newly created checks and change any of their settings.

After a small amount of tinkering, or none at all, we should have all our checks up and running.

For APIs medium and large, and for any that is already described with an OAS file, this check-generating procedure can:

Save us a large amount of time otherwise spent in manual check setup.
Reduce the chance for human error in check configuration.
Help us ensure the right checks are set up according to our single source of truth.

Higher check coverage can ultimately give us higher confidence that our API is working as expected, and basing our monitoring on our OpenAPI specification enables us to get to higher coverage faster - without losing flexibility when building our checks.

Building on our generated monitors

As we look to increase our endpoint coverage across our APIs, we might want to explore the following resources:

Learning more about API monitoring basics and best practices.
Defining our API checks as code to scale our setup.
Integrating our API monitoring with E2E monitoring for our websites.

API Monitoring for the JAMStack

Tim Nolet 👨🏻‍🚀 — Fri, 09 Apr 2021 13:51:51 +0000

Originally published at https://www.checklyhq.com/guides/api-monitoring

Application Programming Interfaces (APIs) are used throughout software to define interactions between different software applications. In this article we focus on web APIs specifically, taking a look at how they fit in the JAMStack architecture and how we can set up API monitoring in order to make sure they don't break and respond fast.

APIs and the JAMStack

With the rise of the JAMStack, the already broadly used web APIs have been brought further into the spotlight and explicitly named as cornerstone of a new way of building web applications. In the JAMStack paradigm, applications rely on APIs (the A in "JAM") returning structured data (JSON or XML) when queried via the HTML and Javascript-based frontend.

The API calls might be aimed at internal services or at third-parties handling complex flows such as content management, authentication, merchant services and more. An example of third-party API could be Stripe, which acts as payment infrastructure for a multitude of businesses.

Given their importance in this new kind of web application, APIs both internal and external need to be tightly monitored, as failures and performance degradations will immediately be felt by the end-user.

API failures

API endpoints can break in a variety of ways. The most obvious examples are:

The endpoint is unresponsive/unreachable.
The response is incorrect.
The response time is too high.

All of the above can result in the application becoming broken for the end-user. This applies to internal APIs and, especially in the case of JAMStack applications, to third parties as well. API checks allow us to monitor both by mimicking the end-user's own behaviour.

API checks

If we were interested in just verifying a server or a virtual machine's availability, we could rely on a simple ping/uptime monitoring solution. API monitoring is more fine-grained than that though, as we need to validate functionality and performance on each API endpoint. API checks do exactly that, and they are composed of the following:

An HTTP request.
One or more assertions, used to specify exactly what the response should look like, and fail the check if the criteria are not met.
A threshold indicating the maximum acceptable response time.

The more customisable the HTTP request is, the more cases can be covered, for example with authentication, headers and payloads.

It is worth noting that in real-world scenarios, requests do not happen in a vacuum: they are often handling data retrieved previously, possibly by earlier API calls. Therefore, some mechanism to gather this data and inject it into the request is often needed.

Let's dive in deeper into each point.

Configurable HTTP request

There is a large variety of valid requests that a user might make to a given endpoint. Being able to customise all aspects of our test request is therefore fundamental. Key aspects are:

Method, like GET, PUT, POST, DELETE, etc
Headers, like Accept, Authorization, Content-Type, Cookie, User-Agent, etc
Query parameters

Essentially, we are trying to craft a complete request for exact endpoint. Not only that, but we might want to have multiple requests set up to cover specific options or negative cases, too.

One such case can be where user-specified parameters such as pagination and timeframes might largely change the response. This is exemplified by the List Customers method in Stripe's Customer API, which we can use to query elements in very different ways, such as by just specifying a limit of results or asking for all results linked to a specific creation date. In this case, both of the following cases are worth monitoring:

curl https://api.stripe.com/v1/customers \
  -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \
  -d limit=3 \
  -G

curl https://api.stripe.com/v1/customers \
  -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \
  -d created=1616519668 \
  -G

If we chose to set up a call using Javascript, for example, we could achieve the same call as in the first case above using axios:

const { default: axios } = require("axios");
const AUTH_TOKEN = Buffer.from(process.env.API_KEY).toString('base64')

axios({
    method: 'get',
    url: 'https://api.stripe.com/v1/customers',
    headers: { 
        'Authorization': `Basic ${AUTH_TOKEN}`,
        'content-type': 'application/x-www-form-urlencoded'
    },
    data: 'limit=3'
}).then((response)=> {
    console.log(response.data)
})

Assertions

To validate the API response, we should be able to check against

Status code
Headers
Body

Let's look at an example: creating a customer via the Stripe Customer API. Since we are not the API's developers, we are assuming the result we get running call right now is correct and can be used to model our assertions. Let's run the following curl command in verbose mode:

curl -v https://api.stripe.com/v1/customers \
  -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \
  -d description="My First Test Customer (created for API docs)"

Within the lengthy output we find the respose (in curl denoted by the '<' symbol), and in it all the important details we need for our assertions.

First, we notice the successful status code:

< HTTP/2 200

After that, we can see the headers, which we might want to check for:

< content-type: application/json
< content-length: 1190
< access-control-allow-credentials: true
< access-control-allow-methods: GET, POST, HEAD, OPTIONS, DELETE
< access-control-allow-origin: *
< access-control-expose-headers: Request-Id, Stripe-Manage-Version, X-Stripe-External-Auth-Required, X-Stripe-Privileged-Session-Required
< access-control-max-age: 300
< cache-control: no-cache, no-store
< request-id: req_S9P5NqvZXzvvS0
< stripe-version: 2019-02-19
< x-stripe-c-cost: 0
< strict-transport-security: max-age=31556926; includeSubDomains; preload

And finally the response body, which we might want to inspect to ensure the right data is being sent back:

{ 
  "id": "cus_JAp37QquOLWbRs",
  "object": "customer",
  "account_balance": 0,
  "address": null,
  "balance": 0,
  "created": 1616579618,
  [clipped]

We could expand on our previous code example by add adding an assertion library, such as chai's or Jest expect:

const { default: axios } = require("axios");
const expect = require('expect')

const AUTH_TOKEN = Buffer.from(process.env.API_KEY).toString('base64')

axios({
    method: 'get',
    url: 'https://api.stripe.com/v1/customers',
    headers: { 
        'Authorization': `Basic ${AUTH_TOKEN}`,
        'content-type': 'application/x-www-form-urlencoded'
    },
    data: 'limit=3'
}).then((response)=> {
    console.log(response.data)
    expect(response.status).toBe(200) // 1) assert again status code 
    expect(response.headers['content-type']).toBe('application/json') // 2) assert against header
    expect(response.data['has_more']).toBe(true) // 3) assert against body
})

We are now asserting against all three point mentioned above. We could of course go on and add additional assertions against both headers and body.

Response time thresholds

Having an endpoint return the correct result is only half the battle. It is imperative that the response reaches the user quickly enough not to disrupt any dependent workflow. In the worst case, where the response time exceeds what the end user is prepared to wait, a performance failure is undistinguishable from a functional one.

The easiest way to handle this requirement would be to assert that the specific response time be lower than a certain value, or even just set a timeout for our axios request by adding the timeout: 7500 property in the previously shown request config.

Instead of simply asserting against a specific response, we might want to set different thresholds: based on the nature of our service, a 2x slowdown might still leave it in what we define as an operational state, while a 10x one might not.

API monitoring best practices

Now that we are clear on the key requirements for setting up API checks, let's think about what and how we should monitor.

Monitor every endpoint

We want to monitor every API endpoint our application exposes. Remember that different HTTP methods define different API endpoints. For example:

GET /user/:id
PUT /user/:id

The above count as two separate endpoints, even though the URL is the same.

Cover key API parameters

Some parameters can change the endpoint's response significantly. We shoud strive to have separate checks verifying that the endpoint is behaving correctly across different configurations.

Keep checks focused & independent

API monitoring checks must be organised as to minimise the time needed to identify resolve the underlying issue. This means we need to keep our checks focused on a specific case (vs trying to have a single check do many things) and independent from each other (vs building chains of checks that build on top of one another).

Scheduled global API checks

Checkly specialises in API monitoring and allows users to run API checks on a schedule from global locations. We can combine these checks with custom alerting to be able to quickly respond and remediate potential API issues.

A Checkly API check is comprised of the following components.

Main HTTP request

The most basic building block of Checkly's API check is the main HTTP request. This can be fully configured in its method, url, parameters and body to fully reproduce a real-world web API call.

Assertions

Assertions allow us to check for every key aspect of the response. A check with one or more failing assertions will enter failing state and trigger any connected alert channel.

In this example, we are checking against:

The status code, expected to be 200.
The id of one of the customers returned as part of the response's JSON body. Here we could assert a specific value, but in this case we are happy with just verifying that the field is not empty.
The value of the Content-Encoding header, expected to equal gzip.

Response time limits

Response time limits enable us to set different thresholds to decide exactly which response time maps to a hard failure, a pass or a degradation. We can use transitions between these states to trigger different kinds of alerts using our preferred channels.

Setup and teardown scripts

Checkly is highly programmable and allows users to run scripts before and after the main HTTP request of an API check.

Setup scripts run before our check and give us access to properties like the URL, headers and query parameters, enabling us to set up all the prerequisites for a successful request. Some examples could be:

Fetching a token from a different API endpoint.
Setting up test data on the target system.
Formatting data to be sent as part of the request.

Teardown scripts run after the request has executed, bur right before the assertions. They are useful for manipulating the response (for example to remove sensitive information) or removing any test data on the target system.

Improving our monitoring

As we increase our monitoring coverage across our APIs, we can also increase the efficacy of our setup by:

Importing existing Swagger/OpenAPI specs or even cURL commands using built-in functionality.
Defining our API checks as code to scale our setup while lowering maintenance needs.
Combining our API checks with E2E monitoring for any website or web app service whose API we might be monitoring.

Banner image: “rover 200 framing line” by spencer_cooper is licensed under CC BY-ND 2.0

End to end monitoring

Tim Nolet 👨🏻‍🚀 — Mon, 29 Mar 2021 18:20:13 +0000

This article originally appeared in Checkly's Guides.

End-to-end monitoring uses headless browser automation tools like Puppeteer and Playwright to continuously test your website's key user flows. This article summarises the most important points on this topic and gets you up and running in 10 minutes.

Headless browser testing

Over the course of the last decade, especially thanks to tools such as Selenium and (more recently) Cypress, automated End-to-End testing (E2E testing) has become widespread across industries.

Broadly speaking, E2E testing entails running fully automated test suites with the goal of catching bugs before they hit production and, therefore, negatively affect the user experience. These test suites need to be carefully scripted using dedicated tools, as well as to be made stable and fast enough to test the most important end-user flows on every build, PR or commit, depending on the application under test and the organisation's automation maturity.

The industry has learned to struggle with the challenges this approach presents:

Long-running suites.
Test flakiness.
Expensive test infrastructure.

All of the above lead to higher costs and slower delivery.

The appearance of mature headless browser automation tools, such as Puppeteer and Playwright, offers a response to many of the above issues by allowing testing in the browser without its GUI, which yields higher speed and stability coupled with lower resource consumption.

E2E monitoring (AKA synthetic monitoring)

While this nimbler, more reliable kind of test is already a big improvement for pre-production testing, it enables a completely new approach in production monitoring: we can now continuously run E2E tests against our production systems. This enables us to have real-time feedback on the status of our website's key user flows from a user's perspective. This is E2E monitoring, also known as synthetic monitoring or active monitoring.

This comes with a significant, often underestimated advantage: it allows us to catch all those things that might break in production that can't be caught during pre-production testing. We are now running directly against the system that the end-user is actually interacting with, and will be able to monitor its behaviour in real time.

What could this look like in practice? Let's look at an e-commerce example.

Monitoring a web shop

A few key flows for an e-commerce websites could be:

Logging in
Finding a product through search
Adding products to the basket and checking out

Let's see how to set them up - for this example, we will do that on our demo web shop.

Playwright E2E tests

Using Playwright, we can script our three E2E scenarios as follows.

Login scenario:

const { chromium } = require("playwright");

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // click on the login button and go through the login procedure
  await page.click("#login");
  await page.type("#n-email", "user@email.com");
  await page.type("#n-password2", "supersecure1");
  await page.click("#goto-signin-btn");

  // wait until the login confirmation message is shown
  await page.waitForSelector("#login-message", { visible: true });

  // close the browser and terminate the session
  await browser.close();
})();

Search scenario:

const { chromium } = require("playwright");
const assert = require("chai").assert;

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  const bookList = [
    "The Foreigner",
    "The Transformation",
    "For Whom the Ball Tells",
    "Baiting for Robot",
  ];

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // search for keyword
  await page.click(".topbar > input");
  await page.type(".topbar > input", "for");
  await page.click("#button-search");
  await page.waitForSelector(
    ".shop-content > ul > .preview:nth-child(1) > .preview-title"
  );

  // halt immediately if results do not equal expected number
  let resultsNumber = (await page.$$(".preview-title")).length;
  assert.equal(resultsNumber, bookList.length);

  // remove every element found from the original array...
  for (i = 0; i < resultsNumber; i++) {
    const resultTitle = await page.$eval(
      `.preview:nth-child(${i + 1}) > .preview-title`,
      (e) => e.innerText
    );

    const index = bookList.indexOf(resultTitle);
    bookList.splice(index, 1);
  }

  // ...then assert that the original array is now empty
  assert.equal(bookList.length, 0);

  // close the browser and terminate the session
  await browser.close();
})();

Checkout scenario:

const { chromium } = require("playwright");

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  const navigationPromise = page.waitForNavigation();

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // add the first item to the cart
  await page.click(`.preview:nth-child(1) > .preview-author`);
  await page.click(".detail-wrapper > .call-to-action");
  await page.click("#logo");

  // wait until navigation is complete
  await navigationPromise;

  // navigate to cart and proceed
  await page.click("#cart");
  await page.click(".cart > .call-to-action");
  await page.click("#s-name");

  // fill out checkout info
  await page.type("#s-name", "Max");
  await page.type("#s-surname", "Mustermann");
  await page.type("#s-address", "Charlottenstr. 57");
  await page.type("#s-zipcode", "10117");
  await page.type("#s-city", "Berlin");
  await page.type("#s-company", "Firma GmbH");
  await page.click(".checkout > form");
  await page.click("#asap");

  // confirm checkout
  await page.click(".checkout > .call-to-action");

  // wait until the order confirmation message is shown
  await page.waitForSelector("#order-confirmation", { visible: true });

  // close the browser and terminate the session
  await browser.close();
})();

These can be run on our own machine without issues with very little preparation with a simple node script.js.

Monitoring application performance

A web application's performance plays a primary role in the user experience it delivers. From the user's perspective, a fully functional application that is not performant quickly becomes indistinguishable from a broken one.

Using Playwright together with browser APIs or additional performance libraries, our end-to-end monitoring setup can be easily extended to include application performance.

Measuring execution time

An effective and granular way to gauge performance is to measure how long our scenario takes to execute. A very simple way to achieve this is to just time our script's execution with time node script.js.

Oftentimes it pays to be more granular. For example, we might want to measure the durations of certain segments of a given flow and assert against them. We can do all this in our script. For example, in the case of our longer checkout example:

const { chromium } = require("playwright");
// we add an assertion library
const assert = require("chai").assert; 

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  const navigationPromise = page.waitForNavigation();

  // get first timestamp
  const tFirstNavigationStarts = Date.now()

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // get second timestamp
  const tFirstNavigationEnds = Date.now()

  // add the first item to the cart
  await page.click('.preview:nth-child(1) > .preview-author');
  await page.click(".detail-wrapper > .call-to-action");
  await page.click("#logo");

  // ...

  // wait until the order confirmation message is shown
  await page.waitForSelector("#order-confirmation", { visible: true });

  // get thirds timestamp
  const tScenarioEnds = Date.now();

  // calculate timings
  const dNavigation = tFirstNavigationEnds - tFirstNavigationStarts;
  const dScenario = tScenarioEnds - tFirstNavigationStarts ;

  // assert against the timings we have measured
  assert.isBelow(dNavigation, 1750, 'Initial navigation took longer than 1.75s')
  assert.isBelow(dScenario, 3000, 'Total scenario took longer than 3s')

  // close the browser and terminate the session
  await browser.close();
})();

We can also use Web Performance APIs such as Navigation Timing and Resource Timing, as well as libraries such as Google Lighthouse. For more examples, see our dedicated performance guide.

End to end application monitoring

Unlike headful tools, headless ones tend to not be very resource-hungry, which makes it easier to move our scripts to the cloud. Checkly runs on top of AWS Lambda, and enables us to quickly copy-paste our script and set it up to run on a schedule from locations around the world.

We can move our scripts to separate checks to keep them independent - we want to optimise for parallelisation and clarity of feedback.

As soon as a check runs red, we are alerted in real time and can intervene before the issue impacts our users. Alerting can be set up with all the industry-standard channels like Pagerduty, Opsgenie, Slack, email, SMS and more.

On-demand checking

Active monitoring and event-triggered testing do not exclude one another. You might want to have checks kicked off every time you deploy to production, or on every merge, PR or commit, or you might also want to run against your staging or development server. The choice has to be made based on your workflow and on your automation strategy.

CI/CD

Tests can be kicked off of CI pipelines. You might want to use different hooks (for e.g. smoke vs regression testing) in different stages and against different targets. Checkly supports all major CI servers.

Develop-preview-test

If you are using provider like Vercel, you can automatically trigger your checks to run on deployed PRs, too, to reap the benefits of the develop-preview-test approach.

Pitfalls

We learned things the hard way so you do not have to. When starting out, keep an eye out for the following pitfalls:

Non-independent tests: tests that rely on one another in any way (e.g. order of execution, test data) are hard to parallelise, resulting in longer execution times and potentially higher flakiness. Keep your tests independent.
Long, unfocused tests: checking too much in a single test will make failures harder to debug. Break it up instead, and enjoy the added parallelisation.
Messing up your own metrics, KPIs: remember, if you are not running against production, you want to make sure your E2E monitoring checks or tests are filtered out of your analytics. This is rather easy to do, with most headless browser tools normally identifying themselves as such from the start.

Banner image: detail from "Outdoor Gas Installation" by christian.senger is licensed under CC BY-SA 2.0

Monitoring-as-Code

Tim Nolet 👨🏻‍🚀 — Tue, 16 Mar 2021 13:28:49 +0000

This article originally appeared in Checkly's Guides.

The trend of declaring infrastructure as code has been picking up steam over the last few years, offering a way for DevOps teams to transparently manage and scale cloud infrastructure. Why should the way we manage monitoring be any different? In this article, we address this point and illustrate it with a practical example of Monitoring-as-Code on Checkly.

Infrastructure-as-Code

Historically, IT infrastructure has been provisioned manually, both on premise and in the cloud. This presented several challenges, including fragmented workflows, lack of transparency and scalability issues. In response to these problems, the last few years have seen a shift to the Infrastructure-as-Code (IaC) paradigm, in which large-scale systems are declared in configuration files as code.

A new generation of tools has emerged to serve this use case, the most notable example of which is HashiCorp Terraform. Terraform provides a CLI workflow allowing users to specify what the final infrastructure setup should look like, and then takes care of all the intermediate steps needed to get there.

Terraform can be used to provision infrastructure on many different cloud vendors thanks to its provider ecosystem: each provider maps to the vendor's API, exposing different resources in a domain-specific language known as HCL.

Monitoring the IaC way

Setting up monitoring can present some of the same issues as provisioning infrastructure. This becomes apparent when we move past the initial rollout or proof-of-concept and onboard multiple products and/or teams, and see our monitoring setup rapidly grow in scope - along with its maintenance needs, that is.

Monitoring-as-Code learns from IaC and brings your monitoring config closer to your application and your development workflows. How? By having it also declared as code, much like you would do with any kind of IT infrastructure.

Why Monitoring-as-Code

What does one gain when moving from a manual to a Monitoring-as-Code approach? The main advantages are:

Better scalability through faster provisioning and easier maintenance.
Better history and documentation: config files can be checked into source control.
Shared monitoring setup visibility (and easier shared ownership) in DevOps teams.

Monitoring-as-Code with Checkly

Users who have just started out will be familiar with creating checks, groups, alert channels and other resources through the Checkly UI. The official Terraform provider enables them to instead declare exactly what their active monitoring setup should look like, and have it provisioned by Terraform in just a few seconds - regardless of whether that means creating tens, hundreds or thousands of resources.

You can find the Checkly Terraform provider on the official Terraform registry.

Monitoring an e-commerce website - as code

How does this all look like in practice? Let's find out by creating a small monitoring setup for our demo e-commerce website.

Setting up our Terraform project

For our example we will be creating browser checks using Playwright scripts we have previously written as part of our Playwright guides.

First will be our login scenario:

const { chromium } = require("playwright");

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // click on the login button and go through the login procedure
  await page.click("#login");
  await page.type("#n-email", "user@email.com");
  await page.type("#n-password2", "supersecure1");
  await page.click("#goto-signin-btn");

  // wait until the login confirmation message is shown
  await page.waitForSelector("#login-message", { visible: true });

  // close the browser and terminate the session
  await browser.close();
})();

...then our search scenario...

const { chromium } = require("playwright");
const assert = require("chai").assert;

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  const bookList = [
    "The Foreigner",
    "The Transformation",
    "For Whom the Ball Tells",
    "Baiting for Robot",
  ];

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // search for keyword
  await page.click(".topbar > input");
  await page.type(".topbar > input", "for");
  await page.click("#button-search");
  await page.waitForSelector(
    ".shop-content > ul > .preview:nth-child(1) > .preview-title"
  );

  // halt immediately if results do not equal expected number
  let resultsNumber = (await page.$$(".preview-title")).length;
  assert.equal(resultsNumber, bookList.length);

  // remove every element found from the original array...
  for (i = 0; i < resultsNumber; i++) {
    const resultTitle = await page.$eval(
      `.preview:nth-child(${i + 1}) > .preview-title`,
      (e) => e.innerText
    );

    const index = bookList.indexOf(resultTitle);
    bookList.splice(index, 1);
  }

  // ...then assert that the original array is now empty
  assert.equal(bookList.length, 0);

  // close the browser and terminate the session
  await browser.close();
})();

...and finally our checkout scenario.

const { chromium } = require("playwright");

(async () => {

  // launch the browser and open a new page
  const browser = await chromium.launch();
  const page = await browser.newPage();

  const navigationPromise = page.waitForNavigation();

  // navigate to our target web page
  await page.goto("https://danube-webshop.herokuapp.com/");

  // add the first item to the cart
  await page.click(`.preview:nth-child(1) > .preview-author`);
  await page.click(".detail-wrapper > .call-to-action");
  await page.click("#logo");

  // wait until navigation is complete
  await navigationPromise;

  // navigate to cart and proceed
  await page.click("#cart");
  await page.click(".cart > .call-to-action");
  await page.click("#s-name");

  // fill out checkout info
  await page.type("#s-name", "Max");
  await page.type("#s-surname", "Mustermann");
  await page.type("#s-address", "Charlottenstr. 57");
  await page.type("#s-zipcode", "10117");
  await page.type("#s-city", "Berlin");
  await page.type("#s-company", "Firma GmbH");
  await page.click(".checkout > form");
  await page.click("#asap");

  // confirm checkout
  await page.click(".checkout > .call-to-action");

  // wait until the order confirmation message is shown
  await page.waitForSelector("#order-confirmation", { visible: true });

  // close the browser and terminate the session
  await browser.close();
})();

Let's start off by creating a brand new folder:

mkdir checkly-terraform-example && cd $_

To keep things easy, we create a subdirectory...

mkdir scripts

...and copy all our scripts from above into separate files, for example login.js.

Next up, we want to create our main.tf file and include the basic configuration as follows:

variable "checkly_api_key" {}

terraform {
  required_providers {
    checkly = {
      source = "checkly/checkly"
      version = "0.8.1"
    }
  }
}

provider "checkly" {
  api_key = var.checkly_api_key
}

We are ready to initialise our project and have the Checkly Terraform provider set up for us. That is achieved by running:

terraform init

After a few seconds, you should see a similar message to the following:

ragog@macpro learn-terraform % terraform init

Initializing the backend...

Initializing provider plugins...
- Finding checkly/checkly versions matching "0.8.1"...
- Installing checkly/checkly v0.8.1...
- Installed checkly/checkly v0.8.1 (signed by a HashiCorp partner, key ID 4E5AC4D95E185A57)

...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

...

Creating our first browser checks

In the same file, right below our initial instructions, we can now add resources one after the other. They will be browser checks based on the Playwright scripts we previously stored in the scripts directory. Here is what each resource could look like:

resource "checkly_check" "login" {

  name                      = "Login E2E"
  type                      = "BROWSER"
  activated                 = true
  should_fail               = false
  frequency                 = 10
  double_check              = true
  ssl_check                 = false
  use_global_alert_settings = true
  locations = [
    "us-west-1",
    "eu-central-1"
  ]

    script = file("${path.module}/scripts/login.js")

}

Search resource:

resource "checkly_check" "search" {

  name                      = "Search E2E"
  type                      = "BROWSER"
  activated                 = true
  should_fail               = false
  frequency                 = 15
  double_check              = true
  ssl_check                 = false
  use_global_alert_settings = true
  locations = [
    "us-west-1",
    "eu-central-1"
  ]

    script = file("${path.module}/scripts/search.js")

}

Checkout resource:

resource "checkly_check" "checkout" {

  name                      = "Checkout E2E"
  type                      = "BROWSER"
  activated                 = true
  should_fail               = false
  frequency                 = 60
  double_check              = true
  ssl_check                 = false
  use_global_alert_settings = true
  locations = [
    "us-west-1",
    "eu-central-1"
  ]

    script = file("${path.module}/scripts/checkout.js")

}

Now that our Terraform project has been initialised and we have added some resources, we can generate a Terraform plan by running terraform plan.

Terraform will determine all the needed changes to be performed to replicate our monitoring configuration on Checkly. In doing so, we will be asked for our Checkly API key, which we can find under our account settings as shown below. Not on Checkly yet? Register a free account and enjoy your free monthly checks!

We can expose this as an environment variable in order to avoid having to copy-paste it all the time: export TF_VAR_checkly_api_key=<YOUR_API_KEY>.

ragog@macpro learn-terraform % terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # checkly_check.checkout will be created
  + resource "checkly_check" "checkout" {
      + activated                 = true
      + degraded_response_time    = 15000
      + double_check              = true
      + frequency                 = 60
      + id                        = (known after apply)
      + locations                 = [
          + "eu-central-1",
          + "us-west-1",
        ]
      + max_response_time         = 30000
      + name                      = "Checkout E2E"
      + script                    = <<-EOT
            const { chromium } = require("playwright");

            ...

Plan: 3 to add, 0 to change, 0 to destroy.

We can now finally apply our changes with terraform apply. We might be asked for one final confirmation in the command prompt, after which we will be greeted by the following confirmation message:

...

checkly_check.checkout: Creating...
checkly_check.login: Creating...
checkly_check.search: Creating...
checkly_check.checkout: Creation complete after 3s [id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]
checkly_check.login: Creation complete after 3s [id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]
checkly_check.search: Creation complete after 4s [id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Logging in to our Checkly account, we will see the dashboard has been populated with our three checks, which will soon start executing on their set schedules.

Monitoring API correctness and performance

Browser checks are now there to keep us informed on the status of our key website flows. What about our APIs, though? Whether they make up the foundation of our service or they are consumed directly by the customer, we need to ensure our endpoints are working as expected. This is easily achieved by setting up API check resources:

resource "checkly_check" "webstore-list-books" {
  name                      = "list-books"
  type                      = "API"
  activated                 = true
  should_fail               = false
  frequency                 = 1
  double_check              = true
  ssl_check                 = true
  use_global_alert_settings = true
  degraded_response_time    = 5000
  max_response_time         = 10000

  locations = [
    "eu-central-1",
    "us-west-1"
  ]

  request {
    url              = "https://danube-webshop.herokuapp.com/api/books"
    follow_redirects = true
    assertion {
      source     = "STATUS_CODE"
      comparison = "EQUALS"
      target     = "200"
    }
    assertion {
      source     = "JSON_BODY"
      property   = "$.length"
      comparison = "EQUALS"
      target     = "30"
    }
  }
}

We can now once more run terraform plan, followed by terraform apply to see the new check on Checkly:

Alerting

Now that we have our checks in place, we want to set up alerting to ensure we are informed as soon as a failure takes place. Alert channels can be declared as resources, just like the checks. Let's add the following to our main.tf file:

resource "checkly_alert_channel" "alert-email" {
  email {
    address = "<YOUR_EMAIL_ADDRESS>"
  }
  send_recovery = true 
  send_failure = true
  send_degraded = false
}

We are setting up things so that we are alerted when our check starts failing, as well as when it recovers. But we still need to decide which checks will subscribe to this channel, and therefore be able to trigger the alerts. This is done by adding the following inside the resource declaration of each check, e.g.:

resource "checkly_check" "login" {

  name                      = "Login E2E"
  type                      = "BROWSER"
  activated                 = true
  should_fail               = false
  frequency                 = 10
  double_check              = true
  ssl_check                 = false
  use_global_alert_settings = true
  locations = [
    "us-west-1",
    "eu-central-1"
  ]

    script = file("${path.module}/scripts/login.js")

  alert_channel_subscription {
    channel_id = checkly_alert_channel.alert-email.id
    activated  = true
  }

}

Going through the usual terraform plan and terraform apply sequence will apply the changes on our Checkly account:

We are now fully up and running with our monitoring-as-code setup. Our checks will run on a schedule, informing us promptly if anything were to go wrong. Rapidly getting to know about failures in our API and key website flows will allow us to react fast and mitigate impact on our users, ensuring a better experience with our product.

You can find the complete setup described in this guide on our dedicated repository.

Expanding our setup

As our setup expands, we might want to deploy additional tools to make our lives easier. We could:

Iterate over existing Playwright scripts and create multiple checks while declaring only one resource.
Group checks together to better handle them in large numbers.
Use code snippets to avoid code duplication and reduce maintenance.
Move your workflow to Terraform Cloud to easily collaborate with your team when managing your Monitoring-as-Code configuration.

banner image: "Electric Grid" by Duncan Rawlinson - Duncan.co is licensed under CC BY-NC 2.0

Generating PDFs (invoices, manuals and more) from web pages using Puppeteer/Playwright

Tim Nolet 👨🏻‍🚀 — Wed, 16 Dec 2020 15:50:14 +0000

This article was originally published on theheadless.dev

Puppeteer and Playwright can be used to create PDFs from webpages. This opens up interesting automation scenarios for tasks such as archiving, generating invoices, writing manuals, books and more.

This article introduces this functionality and shows how we can customise the PDF to fit our needs.

Generating a PDF file

After loading a page, we use the page.pdf() command to convert it to a PDF.

With Puppeteer:

With Playwright:

Note that we need to pass the path option to have the PDF file actually saved to disk.

WARNING:
This feature is currently only supported in Chromium headless in both Puppeteer and Playwright.

Tweaking the result

It is important to take a quick look at the official docs for page.pdf() (Puppeteer or Playwright), as it is almost certain that we will want to tweak the appearance of our page in the resulting PDF.

In certain cases, our webpage might look significantly different in our PDF compared to our browser. Depending on the case, it can pay off to experiment with the following:

We might need to set option printBackground to true in case graphical components appear to be missing in the generated PDF.
By default, page.pdf() will generate a PDF with adjusted colors for printing. Setting the CSS property -webkit-print-color-adjust: exact will force rendering of the original colors.
Calling page.emulateMedia('screen') changes the CSS media type of the page.
Setting either width and height or format to the appropriate value might be needed for the page to be displayed optimally.

Customising header and footer

We can also have custom headers and footers added to our pages, displaying values such as title, page number and more. Let's see how this looks on your favourite website:

With Puppeteer:

With Playwright:

We are including the following template files for our header...

<html>
  <head>
    <style type="text/css">
      #header {
        padding: 0;
      }
      .content {
        width: 100%;
        background-color: #777;
        color: white;
        padding: 5px;
        -webkit-print-color-adjust: exact;
        vertical-align: middle;
        font-size: 15px;
        margin-top: 0;
        display: inline-block;
      }
      .title {
        font-weight: bold;
      }
      .date {
        text-align:right;
      }
    </style>
  </head>
  <body>
    <div class="content">
        <span class="title"></span> -
        <span class="date"></span>
        <span class="url"></div>
    </div>
  </body>
</html>

...and footer:

<html>
  <head>
    <style type="text/css">
      #footer {
        padding: 0;
      }
      .content-footer {
        width: 100%;
        background-color: #777;
        color: white;
        padding: 5px;
        -webkit-print-color-adjust: exact;
        vertical-align: middle;
        font-size: 15px;
        margin-top: 0;
        display: inline-block;
        text-align: center;
      }
    </style>
  </head>
  <body>
    <div class="content-footer">
      Page <span class="pageNumber"></span> of <span class="totalPages"></span>
    </div>
  </body>
</html>

The first page of the generated PDF looks as follows:

TIP:
Chromium sets a default padding for header and footer. You will need to override it in your CSS.

Further considerations

We can easily transform existing web pages into PDF format, just as we have shown in our example. An even more interesting use case is about generating a brand new document: now we can use our existing HTML and CSS skills to produce high-quality PDFs, often eliminating the need for LaTeX or similar tools.

See points 2 and 3 of the following section for practical examples of this approach.

Challenging automation flows

Tim Nolet 👨🏻‍🚀 — Fri, 11 Dec 2020 15:25:35 +0000

This article was originally published on theheadless.dev

While automation tools are fundamental to modern software development, they also have the innate potential to be used for malicious purposes. This applies to Puppeteer and Playwright, too.

As a consequence, some user flows are made purposely hard to automate to defend against threat actors. Some examples:

New user signup
Social login through providers like Google, Facebook etc.
"Forgot password" and similar reset scenarios

There are several means through which automation is made more difficult.

Captchas

Captchas are a popular measure many websites take in order to counter automation for sensitive user flows, with Google's ubiquitous reCAPTCHA being perhaps the most known example. Captchas work by challenging users with tasks that are trivial for humans and hard for bots, like finding objects belonging to the same category in an image or reading obfuscated strings of text.

It is beyond the point of a test to try and solve a captcha: we are not trying to verify that the captcha itself works, rather to get past the captcha so we can test the functionality of our target platform. Also, if we were able to automatically solve a captcha, that would defeat the whole purpose of including one in our application.

For pre-production environments, we can:

Set captcha to always pass
Have captchas removed

Bot detection

Certain platforms might have built-in mechanisms to tell manual and automated traffic apart, with the goal of shutting down the latter. These could be based on traffic volume analysis, HTTP fingerprinting, IP blacklisting or even on user behaviour analysis (e.g. the way the user moves the mouse cursor). Headless browsers will normally identify themselves as such in their User-Agent request headers, making detection trivial:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 
(KHTML, like Gecko) HeadlessChrome/79.0.3945.0 Safari 537.36

Bot detection could be deactivated for pre-production environments in order to allow automation. For production environments, a secret may be included in the User-Agent (or similar mechanism) for the system to recognise test bots and allow them through:

With Puppeteer :

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/79.0.3945.0 Safari 537.36 Secret/<MY_SECRET>');

With Playwright:

const browser = await chromium.launch();
const context = await browser.newContext({
  userAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/79.0.3945.0 Safari 537.36 Secret/<MY_SECRET>'
});
const page = await context.newPage('https://example.com');

Automation-resistant UIs

Certain industries are particularly affected by malicious automation attempts, especially where high-volume automation offers an unfair advantage. Examples of this include online betting and gaming.

In these cases, UI automation is specifically made more difficult by utilising components that are harder to interact with, and in some cases by randomising attribute values for key elements in order to prevent referencing. Even if this measure does not defeat all attempts, it can make it excruciatingly hard to produce reliable automated scripts, which might be enough to discourage a large number of attackers.

Warning:
Whenever looking at production environments, keep in mind that adding exceptions to any of the methods listed above might open new attack vectors for malicious users. Always conduct proper risk analysis before proceeding.

Puppeteer vs Selenium vs Playwright, a speed comparison

Tim Nolet 👨🏻‍🚀 — Fri, 04 Dec 2020 11:37:00 +0000

When we decided to build Checkly's browser checks, we chose to do so with Puppeteer, an open-source headless browser automation tool, later adding Playwright, too. We wanted to support users with synthetic monitoring and testing to let them know whether their websites worked as expected at any given moment. Speed was a primary concern in our case.

Yet, determining which automation tool is generally faster is far from simple. Therefore we decided to run our own benchmarks to see how newcomers Puppeteer and Playwright measured against the veteran WebDriverIO (using Selenium and the DevTools automation protocols).

Among the results of our benchmark were also some unexpected findings, like Puppeteer being significantly faster on shorter scripts and WebDriverIO showing larger than expected variability in the longer scenarios. Read below to know more about the results and how we obtained them.

Table of content

Why compare these automation tools?
Methodology, or how we ran the benchmark

a. General guidelines
b. Technical setup
c. Measurements
d. What we did not measure (yet)
The results

a. Running against a demo website
b. Running against a real-world web application
Conclusion

a. Takeaways

Why compare these automation tools?

A benchmark including Puppeteer/Playwright and Selenium is pretty much an apples-and-oranges comparison: these tools have significantly different scopes, and anyone evaluating them should be aware of their differences before speed is considered.

Still, most of us having worked with Selenium for many years, we were keen to understand if these newer tools were indeed any faster.

It is also important to note that WebDriverIO is a higher-level framework with plenty of useful features, which can drive automation on multiple browsers using different tools under the hood.

Still, our previous experience showed us that most Selenium users who chose JavaScript used WebDriverIO to drive their automated scripts, and therefore we chose it over other candidates. We were also quite interested in testing out the new DevTools mode.

Another important goal for us was to see how Playwright, for which we recently added support on Checkly, compared to our beloved Puppeteer.

Methodology, or how we ran the benchmark

Feel free to skip this section in case you want to get straight to the results. We still recommend you take the time to go through it, so that you can better understand exactly what the results mean.

General guidelines

A benchmark is useless if the tools being tested are tested under significantly different conditions. To avoid this, we put together and followed these guidelines:

Resource parity: Every test was run sequentially on the same machine while it is "at rest", i.e. no heavy workloads were taking place in the background during the benchmark, potentially interfering with the measurements.
Simple execution: Scripts were run as shown in the "Getting started" documentation of each tool, e.g. for Playwright: node script.js and with minimal added configuration.
Comparable scripts: We strived to minimise differences in the scripts that were used for the benchmark. Still, some instructions had to be added/removed/tweaked to achieve stable execution.
Latest everything: We tested all the tools in their latest available version at the time of the publishing of this article.
Same browser: All the scripts ran against headless Chromium.

See the below section for additional details on all points.

Technical setup

For each benchmark, we gathered data from 1000 successful sequential executions of the same script.

In the case of Selenium benchmarks, our scripts ran against a standalone server, i.e. we did not start a new server from scratch for each run (even though we always used clean sessions), as some frameworks do. We made this choice to limit overhead on execution time.

We ran all tests on the latest-generation MacBook Pro 16" running macOS Catalina 10.15.7 (19H2) with the following specs:

Model Identifier: MacBookPro16,1
Processor Name: 6-Core Intel Core i7
Processor Speed: 2,6 GHz
Number of Processors: 1
Total Number of Cores: 6
L2 Cache (per Core): 256 KB
L3 Cache: 12 MB
Hyper-Threading Technology: Enabled
Memory: 16 GB

We used the following dependencies:

bench-wdio@1.0.0 /Users/ragog/repositories/benchmarks/scripts/wdio-selenium
├── @wdio/cli@6.9.1
├── @wdio/local-runner@6.9.1
├── @wdio/mocha-framework@6.8.0
├── @wdio/spec-reporter@6.8.1
├── @wdio/sync@6.10.0
├── chromedriver@87.0.0
└── selenium-standalone@6.22.1
scripts@1.0.0 /Users/ragog/repositories/benchmarks/scripts
├── playwright@1.6.2
└── puppeteer@5.5.0

You can find the scripts we used, together with the individual results they produced, in the dedicated GitHub repository.

Measurements

Our results will show the following values, all calculated across 1000 runs:

Mean execution time (in seconds)
Standard deviation (in seconds): A measure of the variability in execution time.
Coefficient of variation (CV): A unitless coefficient showing the variability of results in relation to the mean.
P95 (95th-percentile measurement): The highest value left when the top 5% of a numerically sorted set of collected data is discarded. Interesting to understand what a non-extreme but still high data point could look like.

What we did not measure (yet):

Reliability: Unreliable scripts quickly get useless, no matter how fast they execute.
Parallelisation efficiency: Parallel execution is very important in the context of automation tools. In this case, though, we first wanted to understand the speed at which a single script could be executed.
Speed in non-local environments: Like parallelisation, cloud execution is also an important topic which remains outside of the scope of this first article.
Resource usage: The amount of memory and computing power needed can determine where and how you are able to run your scripts.

Stay tuned, as we might explore these topics in upcoming benchmarks.

The results

Below you can see the aggregate results for our benchmark. You can find the full data sets in our GitHub repository.

Running against a demo website

Our first benchmark ran against our demo website. Hosted on Heroku, this web page is built using Vue.js and has a tiny Express backend. In most cases, no data is actually fetched from the backend, and the frontend is instead leveraging client-side data storage.

In this first scenario, performing a quick login procedure, we expected an execution time of just a few seconds, great for highlighting potential differences in startup speed between the actual tools.

The aggregate results are as follows:

The first thing that catches one's attention is the large difference between the average execution time for Playwright and Puppeteer, with the latter being almost 30% faster and showing less variation in its performance. This left us wondering whether this was due to a higher startup time on Playwright's side. We parked this and similar question to avoid scope creep for this first benchmark.

The second surprise was the lower overall variability shown in the WebDriverIO runs. Also interesting is just how close the results are: the chart shows the lines crossing each other continuously, as the automation protocol does not seem to make a sizeable difference in execution time in this scenario.

Less surprising is perhaps that running Puppeteer without any added higher-level framework helps us shave off a significant amount of execution time on this very short script.

Running against a real-world web application

Previous experience has taught us that the difference between a demo environment and the real world gets almost always underestimated. We were therefore very keen to have the benchmarks run against a production application. In this case we chose our own, which runs a Vue.js frontend and a backend which heavily leverages AWS.

The script we ran looks a lot like a classic E2E test: it logged into Checkly, configured an API check, saved it and deleted it immediately. We were looking forward to this scenario, but each of us had different expectations on what the numbers would look like.

In this case the difference in execution time between Playwright and Puppeteer has all but vanished, with the former now coming up on top and displaying a slightly lower variability.

Proportionally, the difference between the newer tools and both flavours of WebDriverIO is also lower. It is worth noting that the latter two are now producing more variable results compared to the previous scenario, while Puppeteer and Playwright are now displaying smaller variations.

Interestingly enough, our original test for this scenario included injecting cookies into a brand new session to be able to skip the login procedure entirely. This approach was later abandoned as we encountered issues on the Selenium side, with the session becoming unresponsive after a certain number of cookies had been loaded. WebDriverIO handled this reliably, but the cookie injection step exploded the variability in execution time, sometimes seemingly hanging for longer than five seconds.

We can now step back and compare the execution times across scenarios:

Have doubts about the results? Run your own benchmark! You can use our benchmarking scripts shared above. Unconvinced about the setup? Feel free to submit a PR to help make this a better comparison.

Conclusion

First off, let us rank the tools from fastest to slowest for both testing scenarios:

This first benchmark brought up some interesting findings:

Even though Puppeteer and Playwright sport similar APIs, Puppeteer seems to have a sizeable speed advantage on shorter scripts (close to 30% in our observations).
Puppeteer and Playwright scripts show faster execution time (close to 20% in E2E scenarios) ** compared to the Selenium and DevTools WebDriverIO flavours**.
With WebDriverIO, WebDriver and DevTools automation protocols showed comparable execution times.

Takeaways

When running lots of quicker scripts, if there is no need to run cross-browser, it might be worth to run Puppeteer to save time. On longer E2E scenarios, the difference seems to vanish.
It pays off to consider whether one can run a more barebones setup, or if the convenience of WebDriverIO's added tooling is worth waiting longer to see your results.
Fluctuations in execution time might not be a big deal in a benchmark, but in the real world they could pile up and slow down a build. Keep this in mind when choosing an automation tool.
Looking at the progress on both sides, we wonder if the future will bring DevTools to the forefront, or if WebDriver will keep enjoying its central role in browser automation. We suggest keeping an eye on both technologies.

Speed is important, but it can't tell the whole story. Stay tuned, as we surface new and practical comparisons that tell us more about the tools we love using.

banner image: "Evening View of Takanawa". Utagawa Hiroshige, 1835, Japan. Source

Intro to measuring page performance with Puppeteer & Playwright

Tim Nolet 👨🏻‍🚀 — Thu, 26 Nov 2020 13:30:19 +0000

This article was originally published on theheadless.dev

The need for fast and responsive applications has never been greater because of the move from desktop to mobile. Still, web applications have been increasing in complexity and size, with rising load times. It is therefore clear why the topic of webpage performance is more popular today than it likely ever was.

This article aims at giving a practical introduction to the whys and hows of web performance, without getting lost in the depth or breadth of this massive topic.

Why performance matters

The time it takes for a service to become usable, as well as its general responsiveness, bear a lot of weight on the user's perception of that service. Helpful features, great design and other prominent characteristics all become irrelevant when an online service is so slow that users navigate away.

You can build the best web application in the world, but be mindful that each user will have a specific amount of time they are willing to invest in your service to solve their problems. Exceed that amount, and you risk losing them to a different, more performant solution. This is even truer for new users, who haven't yet been given proof of the quality of your service, and are essentially investing their time up-front, hoping for a return.

A competitive differentiator

There is a brighter side to the topic: if low performance can sink an online platform, high performance can very well help it rise to the top. Speed and responsiveness can be a differentiating characteristic for a service, prompting users to choose it over the competition. Therefore an investment in this area will almost always pay off. Some notorious real-world examples from known businesses include:

Pinterest decreasing wait time for their users, increasing both traffic and conversions.
Zalando applying small improvements in load time and finding a direct correlation with increased revenue per session.
The BBC discovering that every extra second that a page took to load led to 10% of users leaving the page.

Measuring performance

Given the importance of page performance, it is no coincidence that browsers expose a ton of insights into performance metrics. Being aware of how your application scores against these across time will provide you the feedback you need to keep it performant for your users. There are several approaches that can be combined to achieve the best results:

Real user monitoring to understand what performance actual end-users of your service are experiencing.
Synthetic monitoring to proactively gather intel on service performance, as well as to find issues before users stumble into them.
Performance testing to avoid releasing performance regression to production in the first place.
Regular audits to get an overview of your page's performance and suggestions on how to improve it, e.g. with tools such as Google Lighthouse.

Performance with headless tools

As much as we should be striving to build performant applications, we should commit to monitoring and testing performance to enable continuous feedback and rapid intervention in case of degradation. Puppeteer and Playwright give us a great toolkit to power both synthetic monitoring and performance testing.

Access to the Web Performance APIs, especially PerformanceNavigationTiming and PerformanceResourceTiming.
Whenever testing against Chromium, access to the Chrome DevTools Protocol for traffic inspection, network emulation and more.
Easy interoperability with performance libraries from the Node.js ecosystem.

Web Performance APIs

The Navigation Timing and the Resource Timing performance APIs are W3C specifications. The MDN docs very clearly define the scope of both:

Navigation timings are metrics measuring a browser's document navigation events. Resource timings are detailed network timing measurements regarding the loading of an application's resources. Both provide the same read-only properties, but navigation timing measures the main document's timings whereas the resource timing provides the times for all the assets or resources called in by that main document and the resources' requested resources.

We can use the Navigation Timing API to retrieve timestamps of key events in the page load timeline.

With Puppeteer:

With Playwright:

The Resource Timing API allows us to zoom in to single resources and get accurate information about how quickly they are being loaded. For example, we could specifically look at our website's logo:

With Puppeteer:

With Playwright:

Chrome DevTools for performance

The Chrome DevTools Protocol offers many great performance tools for us to leverage together with Puppeteer and Playwright.

One important example is network throttling, through which we can simulate the experience of users accessing our page with different network conditions.

With Puppeteer:

With Playwright:

The DevTools Protocol is quite extensive. We recommend exploring the documentation and getting a comprehensive overview of its capabilities.

Additional performance libraries

Lighthouse can easily be used programmatically with Playwright and Puppeteer to gather values and scores for different metrics, like Time To Interactive (TTI):

With Puppeteer:

With Playwright:

Keeping tests valuable

Tim Nolet 👨🏻‍🚀 — Fri, 20 Nov 2020 11:10:32 +0000

Headless browsers can be leveraged for testing in a variety of ways, and different scenarios do command different approaches. That being said, there are some general pointers most should follow in order to keep their tests valuable.

Here, we define valuable as sustainably expressing meaningful, truthful information about the state of a system. A test that does not reliably fulfill these criteria should be fixed, if possible, or simply removed.

Keep tests short

If they run against a real-world product with a UI that is evolving over time, scripts will need to be regularly updated. This brings up two important points:

Most scripts are not write-and-forget, so each script we write is one more script we will have to maintain.
Like all cases where code and refactoring are involved, how we write scripts can have a significant influence on how long this maintenance effort takes.

Taking example from good software engineering practices, our scripts should strive for simplicity, conciseness and readability:

Simplicity: keep in mind the goal of the script, and keep away from overly complex solutions whenever possible.
Conciseness: simply put, do not be overly verbose and keep scripts as short as they can be.
Readability: follow general best practices around writing code that is easy to read.

The faster we can read and understand a script we (or a teammate) wrote in the past, the quicker we can interpret its results and get to work on updating it and making it relevant again.

TIP: Do not underestimate the compounding effect of having messy scripts across a large test base.

Keep tests focused

Automated tests are effective if they:

Correctly verify the status of the target functionality.
Return within a reasonable amount of time.
Produce a result that can be easily interpreted by humans.

The last point is often overlooked. Scripts by themselves have no meaning if their results mean nothing to whoever is looking at them. Ideally, we want the opposite: interpreting a test success or failure should be close to instantaneous and give us a clear understanding of what is working and what is not.

Oftentimes this is impeded by the tendency to have tests do too much. We can draw an example from two scenarios running against our test site: E2E Checkout and E2E Coupon. While these two have part of their flow in common, and we might be tempted to combine them to avoid a certain degree of duplication, merging them into a single test would obfuscate the meaning of a test failure as we would be testing two different features. If that combined test were to run red, we would be unable to tell whether the entire checkout is not working as expected, or whether users are just unable to redeem coupon codes. Unless we were to devote additional time to diving deep into the failure — which is exactly what we are trying to avoid.

We can avoid this pitfall by making sure our tests are verifying only one feature each.

TIP: Always check the assertions in your test: if they are spanning more than one feature, you would likely be better off splitting your test into multiple different ones.

Keep tests independent

In an effort to remove duplication, tests are often made dependent on the previous execution of one or more other tests. An example could be the following sequence of tests:

test1: 
- creates user1 
- asserts user1 has been created 
test2: 
- logs in with user1 
- asserts user1 has logged in successfully 
test3: 
- goes through checkout as user1 
- asserts that checkout for user1 was successful

In this case, the success of each subsequent test depends on previous tests as much as on its own assertions. In other words, previous tests are driving a test’s success criteria. This form of coupling can create headaches for us in the shape of:

Lower test result readability: we might need to backtrack and look into previous tests when trying to understand what a subsequent one is doing
Harder maintenance: changes might need to be applied across different scripts
Lack of flexibility and parallelisation: tests need to run sequentially in a specific order to work

Having each test encapsulate all it needs to give us a meaningful answer is fundamental to avoid the above issues. In case of our example, a solution could look like:

test1:
- creates user1
- asserts user1 has been created
test2:
- (setup: creates user1)
- logs in with user1
- asserts user1 has logged in successfully
test3:
- (setup: creates user1)
- (setup: logs in with user1)
- goes through checkout as user1
- asserts that checkout for user1 was successful

Where:

Tests can be run in any order, even at the same time.
In the interest of code quality, we would be factoring out setup phases in order to easily reuse them across tests. Jest’s beforeEach and beforeAll hooks are examples of useful helper functions that can help us achieve that.

TIP: If the system you are testing allows it, provision/deprovision your users through APIs in your setup and teardown phases to save time and increase test reliability.

Takeaways

Tests should be reliable and informative in order to be useful.
Keep tests short and focused on testing one feature.
Keep tests independent to maximise their parallelisation potential and reduce total runtime.

DEV Community: Tim Nolet 👨🏻‍🚀

Avoiding hard waits in Playwright and Puppeteer

The problems with hard waits

How to fix it

Built-in waits

Explicit waits

Waiting on navigations and network conditions

Waiting for an element

Waiting on page events

Waiting on page functions

Takeaways

Migrating from Puppeteer to Playwright

Puppeteer and Playwright today

Why switch

What to change in your scripts - short version

What to change in your scripts - in depth

Require Playwright package

The browser context

Waiting

Setting viewport

Typing

Cookies

XPath selectors

Device emulation

File download

File upload

Request interception

New possibilities to be aware of

New selector engines

Saving and reusing state

Locator API

Playwright Inspector

Playwright Test

Trace Viewer

Test Generator

Switching to Playwright for richer browser check results

Debugging Puppeteer & Playwright scripts

Awareness comes first

The error message

Gaining visibility

Increasing logging

Accessing DevTools

The Playwright Inspector

Further reading

OpenAPI/Swagger Monitoring

The OpenAPI Specification

Swagger vs OpenAPI

Exploring the specification

Generating documentation from OpenAPI

Generating boilerplate code from OpenAPI

Generated monitoring checks with Checkly

Building on our generated monitors

API Monitoring for the JAMStack

APIs and the JAMStack

API failures

API checks

Configurable HTTP request

Assertions

Response time thresholds

API monitoring best practices

Monitor every endpoint

Cover key API parameters

Keep checks focused & independent

Scheduled global API checks

Main HTTP request

Assertions

Response time limits

Setup and teardown scripts

Improving our monitoring

End to end monitoring

Headless browser testing

E2E monitoring (AKA synthetic monitoring)

Monitoring a web shop

Playwright E2E tests

Monitoring application performance

Measuring execution time

End to end application monitoring

On-demand checking

CI/CD

Develop-preview-test