In this article we present an overview on how to deal with asynchrony when performing end-to-end tests, using Puppeteer as a web scraper and Jest as an assertion library. We will learn how to automate user action on the browser, wait for the server to return data and for our application to process and render it, to actually retrieving information from the website and comparing it to the data to see if our application actually works as expected for a given user action.
So you got your wonderful web application up and running, and the time for testing has come.... There are many types of test, from Unit tests where you test the individual components that compound your application, to Integration tests where you test how these components interact with eachother. In this article we are gonna talk about yet another type of tests, the End-To-End (e2e) tests.
End-to-end tests are great in order to test the whole application as a user perspective. Tha means testing that the outcome, behavior or data presented from the application is as expected for a given user interaction with it. They test from the front-end to the back-end, treating the application as a whole and simulating real case scenarios. Here it is a nice article talking about what e2e tests are and their importance.
To test javascript code, one of the most common frameworks for assertions is Jest, which allows you to perform all kinds of comparisons for your functions and code, and even testing React components. In particular, to perform e2e tests, a fairly recent tool, Puppeteer, comes to the rescue. Basically it is a web scraper based on Chromium. According to the repo, it is a "Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol".
It provides some methods so you can simulate the user interaction on a browser via code, such as clicking elements, navigating through pages or typing on a keyboard. As having a highly trained monkey perform real case tasks on your application 🐒
You can find the github repos of both testing libraries here:
🃏 Delightful JavaScript Testing
👩🏻💻 Developer Ready: A comprehensive JavaScript testing solution. Works out of the box for most JavaScript projects.
🏃🏽 Instant Feedback: Fast, interactive watch mode only runs test files related to changed files.
See more on jestjs.io
Table of Contents
- Getting Started
- Running from command line
- Additional Configuration
- Documentation
- Badge
- Contributing
- Credits
- License
Getting Started
Install Jest using yarn:
yarn add --dev jest
Or npm:
npm install --save-dev jest
Note: Jest documentation uses yarn commands, but npm will also work. You can compare yarn and npm commands in the yarn docs, here.
Let's get started by writing a test…
Puppeteer
API | FAQ | Contributing | Troubleshooting
Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium.
What can I do?
Most things that you can do manually in the browser can be done using Puppeteer! Here are a few examples to get you started:
- Generate screenshots and PDFs of pages.
- Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)).
- Automate form submission, UI testing, keyboard input, etc.
- Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
- Capture a timeline trace of your site to help diagnose performance issues.
- Test Chrome Extensions.
Give it a spin: https://try-puppeteer.appspot.com/
Getting Started
Installation
To use Puppeteer in your…
Given so, the Puppeteer + Jest become has become a nice, open source way of testing web applications, by opening the application on the headless browser provided by puppeteer and simulating user input and/or interaction, and then checking that the data presented and the way our application reacts to the different actions is as expected with Jest.
In this article we will not cover the whole workflow of testing with puppeteer + jest (such as installing, or setting the main structure of your tests, or testing forms), but focus on one of the biggest things that we have to take into account when doing so: Asynchrony.
🤘 But hey, here you have a great tutorial on how to test with Puppeteer + Jest, to get you started.
Because almost all of the web applications contain indeed some sort of asynchrony. Data is retrieved from the back-end, and that data is then rendered on screen. However, Puppeteer performs all the operations sequentially, so... how can we tell him to wait until asynchronous events have happened?
Puppeteer offers you a way to wait for certain things to happen, using the waitFor functions available for the Page class. The changes you can track are visual changes that the user can observe. For instance you can see when something in your page has appeared, or has changed color, or has disappeared, as a result of some asynchronous call. Then one would compare these changes to be what you would expect from the interaction, and there you have it. How does it work?
Waiting, Waiting, Waiting...⏰
The waitFor function set in Puppeteer helps us deal with asynchrony. As these functions return Promises, usually the tests are performed making use of the async/await ES2017 feature. These functions are:
-
waitForNavigation
await page.waitForNavigation({waitUntil: "networkidle2"});
Whenever a user action causes the page to navigate to another route, we sometimes have to wait a little bit before all the content is loaded. For that we have the waitForNavigation function. it accepts a options object in where you can set a timeout (in ms), or waitUntil some condition is met. The possible values of waitUntil are (according to the wonderful puppeteer documentation):
-
load (default): consider navigation to be finished when the
loadevent is fired. -
domcontentloaded: consider navigation to be finished when the
DOMContentLoadedevent is fired. -
networkidle0: consider navigation to be finished when there are no more than 0 network connections for at least
500ms. -
networkidle2: consider navigation to be finished when there are no more than 2 network connections for at least
500ms.
Tipically, you will want to wait until the whole page is loaded ({waitUntil: load}), but that does not guarantee you that everything is operative. What you can do is wait for a specific DOM element to appear that assures you that the whole page is loaded. you can do that with the following function:
waitForSelector
This function waits for a specific CSS selector to appear, indicating that the element it matches to is on the DOM.
await page.waitForSelector("#create-project-button");
await page.goto("https://www.example.com",{waitUntil: "load"});
await page.click("#show-profileinfo-button"); //Triggers a navigation
/*
We can either wait for the navigation or wait until a selector that indicates
that the next page is operative appears
*/
await page.waitForNavigation({waitUntil: "load"});
await page.waitForSelector("#data-main-table");
waitForFunction
The last one waits for some function to return true in order to proceed. It is commonly used to monitor some property of a selector. It is used when the selector is not appearing on the DOM but some property of it changes (so you cannot wait for the selector because it is already there on the first place). It accepts a string or a closure as arguments.
For example, you want to wait until a certain message changes. The testing would be performed by first getting the message using the evaluate() Puppeteer function. Its first parameter is a function which is evaluated in the browser context (as if you were typing into the Chrome console).We then perform the asynchronous operations that change the message (clicking a button or whatever 🖱🔨), and then waiting for the message to change.
const previousMessage = await page.evaluate( () => document.querySelector('#message').innerHTML);
//Async behaviour...
await page.waitForFunction(`document.querySelector('#message').innerHTML !== ${previousMessage}`); //Wait until the message changes
Using these waitFor functions we can detect when something in our page changes after an async operation, now we just need to retrieve the data we want to test.
Retrieving data from selectors after an Async operation
Once we have detected the changes caused by our asynchronous code, we tipically want to extract some data from our application that we can later compare to the expected visual result from a user interaction. We do that using evaluate() .The most common cases that you face when retrieving data are:
- Checking that a DOM element has appeared
A pretty common case is checking that a given element has been rendered on the page, hence appearing on the DOM. For instance, after saving a post, you should find it in the saved posts section. Navigating there and querying if indeed the DOM element is there is the basic type of data that we can assert (as a boolean assertion).
Find below an example of checking if a given post with an id post-id, where the id is a number we know, is present on the DOM. First we save the post, we wait for the post to be saved, go to the saved posts list/route and see that the post is there.
const id = '243';
await page.click(`#post-card-${id} .button-save-post`);
//The class could be added when the post is saved (making a style change on the button)
await page.waitForSelector(`#post-card-${id} .post-saved`);
await page.click('#goto-saved-posts-btn');
await page.waitForNavigation();
const post = await page.evaluate(id => {
return document.querySelector(`#post-${id}`) ? true : false
},id);
expect(post).toEqual(true);
In there, we can observe a couple of things.
The aforemenctioned need to have unique id's for the tested elements. Given so, querying the selector is way easier and we do not need to do nested queries that get the element based on its position on the DOM (Hey, get me the first tr element from the first row of that table 🙋🏼).
We see how we can pass arguments to the evaluate function and use it to interpolate variables into our selectors. As the fucntion is being evaluated in another scope, you need to bind the variables from node to that new scope, and you can do that via that second parameter.
- Checking for matching property values (e.g innerHTML, option...)
Now imagine that instead of checking that an element is on the DOM, you actually want to check if the list of saved posts rendered on the page actually are the posts you have saved. That is, you want to compare an array of strings with the post names, e.g ["post1,"post2"], with the saved posts of a user (which you can know beforehand for a test user, or retrieve from the server response).
For doing that, you need to query all the title elements of the posts and obtain a given property from them (as it could be their innerHTML, value, id...). After that, you have to convert that information to a serializable value (the evaluate function can only return serializable values or it will return null, that is, always return arrays, strings or booleans, for instance, not HTMLElements...).
An example performing that testing would be:
const likedPosts = ["post1","post2","post3"];
const list = await page.evaluate(() => {
let out = []
/*
We get all the titles and get their innerHTML. We could also query
some property e.g title.getAttribute('value')
*/
const titles = document.querySelectorAll("#post-title");
for(let title of titles){
out.push(title.innerHTML)
}
return out;
});
expect(list).toEqual(likedPosts);
Those are the most basic cases (and the ones you will be using most of the time) for testing the data of your application.
-Asserting that the waitFor is succesful
Another thing you can do instead of evaluate() is, in case you just want to assert a boolean selector or a particular DOM change, is just assign the waitFor() call to a variable and check if it is true. The downside of that method is that you will have to set an estimated timeout to the function that is less than the Jest timeout set at the start. ⏳
If that timeout is exceeded the test will fail. It requires you to put an estimate timeout that you think is enough for the element to be rendered on your page after the request is made (Hmm yeah, I think that around 3 seconds should be enough... 🤔).
For example, we want to check if a new tag has been added to a post querying the number of tag elements present before and after adding tags, that is, comparing their length and see if it has increased, denoting that the tag has indeed been added.
const previousLength = await page.evaluate(() => document.querySelectorAll('#tag').length);
//Add tag async operation...
//Wait maximum 5 seconds to see if the tag has been added.
const hasBeenAdded = await page.waitForFunction(previousLength => {
return document.querySelector('#tag')length > previousLength
}, {timeout: 5000}, previousLength);
expect(hasChanged).toBeTruthy();
Note that you also have to bind the variables to the waitForFunction() as the third element if you specify the waitForFunction parameter as a closure.
In order to get the data to compare with the information we have retrieved from the page, i.e our ground truth, an approach is to have a controlled test user in which we know what to expect for each one of the tests (number of liked posts, written posts). In this approach we can then hardcode 😱 the data to expect, such as the post titles, number of likes, etc... like we did on the examples of the previous section
You can also fake the response data from the server. That way you can test that the data obtained from the back-end is consistent with what is rendered into the application by responding predictable, inmutable data which you know beforehand. This serves for testing if the application responds predictably (parses correctly) the data returned from the server for a given call.
On the next section we will see how to hijack the requests and provide custom data which you know. Puppeteer provides a method to achieve that, but if you want to dwelve more into faking XMLHttpRequest and pretty much all the data your test manages, you should take a look into Sinon.js 💖
Sinon.JS
Standalone and test framework agnostic JavaScript test spies, stubs and mocks (pronounced "sigh-non", named after Sinon, the warrior)
Compatibility
For details on compatibility and browser support, please see COMPATIBILITY.md
Installation
via npm
$ npm install sinon
or via sinon's browser builds available for download on the homepage. There are also npm based CDNs one can use.
Usage
See the sinon project homepage for documentation on usage.
If you have questions that are not covered by the documentation, you can check out the sinon tag on Stack Overflow or drop by #sinon.js on irc.freenode.net:6667.
You can also search through the Sinon.JS mailing list archives.
Goals
- No global pollution
- Easy to use
- Require minimal “integration”
- Easy to embed seamlessly with any testing framework
- Easily fake any interface
- Ship with ready-to-use fakes for XMLHttpRequest, timers and more
Contribute?
See CONTRIBUTING.md for details on how you can contribute to Sinon.JS
…Intercepting Requests and faking Requests with Puppeteer
Imagine that you want to check if the list of saved posts rendered on the page is indeed correct given a list of saved posts for a certain user that we can obtain from an endpoint called /get_saved_posts. To enable requestInterception on puppeteer we just have to set when launching puppeteer
await page.setRequestInterceptionEnabled(true);
With that, we can set all the request to intercept and mock the response data. Of course, that requires knowing the structure of the data returned by the back-end. Tipically, one would store all the fake data response objects into a separate class and then, on request interception, query the endpoint called and return the corresponding data. This can be done like that, using the page.on() function:
Disclaimer: For the API we assume the somewhat typical format https://api.com/v1/endpoint_name, so the parsing to retrieve the endpoint is specific to that format for exemplification purposes, you can detect the Request made based on other parameters of course, your choice 💪
const responses = {
"get_saved_posts": {
status: 200,
//Body has to be a string
body: JSON.stringify({
data: {
posts: ["post1","post2"]
}
});
}
}
page.on('request', interceptedRequest => {
const endpoint = interceptedRequest.url.split('/').pop();
if(responses[endpoint]){
request.respond(responses[endpoint]);
}else{
request.continue();
}
});
You can see the full documentation for the Request class here
One can easily see that, depending on the size of your API, this can be quite complex, and also one of the charms of the e2e testing is to also test if the back-end is giving the correct information and data 😵.
All these methods require for you to enter known data, hardcoded as a response or as a variable to compare. Another way to get the data to compare is to intercept the response and store the data into the variable.
Getting the data to assert from the Response
You can also intercept responses and get the data from there. For instance, we can intercept the get_saved_posts response and store all the posts into a variable.
const posts = [];
page.on('response', response => {
/*
Grab the response for the endpoint we want and get the data.
You could then switch the endpoint and retrieve different data
from the API, such as the post id from before, remember?
*/
const endpoint = response.url().split('/').pop();
if(endpoint === "get_saved_posts"){
const responseBody = await response.json();
posts = responseBody.data.posts
}
})
✍🏼 Note: The page.on() methods for request and response are tipically placed on the beforeAll() method of your tests, after declaring the page variable, as they define a global behavior of your test.
So after your application has rendered everything you can query the DOM elements and then compare with the posts variable to see that your app effectively renders everything as expected.
Summary
In this article we provided a comprehensive overview on how to test applications that present asynchronous data fetched from a server, by using Puppeteer + Jest.
We have learned how to implement waiting for certain events to happen (e.g DOM mutations), that trigger visual changes caused by our asynchronous data. We have gone through the pipeline of detecting, querying and comparing those changes with known data so we can assess that the application works as expected from a user perspective.
Top comments (4)
Great article @Albert. One question around something that's driving me crazy, the following line always ends with a timeout error. Without the
awaitclause is not trustable for the asynchronous behavior.Any thoughts?
Great writing! Thx Albert 🙂
Thank you! I have recently switched to Cypress (cypress.io/) which makes dealing with Asynchrony so much easier :)
I have experience with Jest and Unit tests for a UI Kit, but now I want to write e2e tests for an end-user application with SSR and SPA parts. I thought that I could just use Cypress, but after some research, I've started to doubt:
Now I think about using Jest with Puppeteer for e2e tests and something for screenshot diffs. I need to run my tests in GitlabCI at first.
What would you suggest? Is Cypress still better choice for me?