DEV Community

Cover image for Await, do not make your E2E tests sleep ⏳
Stefano Magni
Stefano Magni

Posted on • Edited on

Await, do not make your E2E tests sleep ⏳

#testing #endtoendtesting #uitesting #performance

I'm working on a big UI Testing Best Practices project on GitHub, I share this post to spread it and have direct feedback.

When testing your UI, you define a sort of key points the app must pass through.
Reaching these key points is an asynchronous process because, almost 100% of the times, your UI does not update synchronously. Those key points are called deterministic events, as known as something that you know that must happen.
It depends on the events you define and how your UI reaches them but, usually, there are some "long" waitings, like XHR requests, and some faster ones, like re-render updates.

The solution to the asynchronous updates seems handy: sleeping/pausing the test for a bunch of milliseconds, tenths of a second, or even seconds. It can make your test working because it gives the app the time to update itself and moving to the next deterministic event to be tested.

Consider that, except for specific and known waitings (like when you use
setInterval or setTimeout), it's totally unpredictable how much the sleeping time should be because it could depend on:

  • the network state (for XHR requests)
  • the total amount of available machine resources (CPU, RAM, etc.)
    • a CI pipeline can limit them for example
    • other apps can consume them on your local machine too
  • the concurrence of other resource-consuming updates (canvas, etc.)
  • a bunch of unpredictable game players like Service Workers, cache management etc. that can make the UI update process faster or slower

Every fixed delay leads your test to be more brittle and increasing its
duration. You are going to find a balance between false negatives—when the test fails because of a too low sleeping time—and exaggerate test
duration.
What about waiting just the right amount of time? The amount of time that makes the test as fast as possible!

Waitings fall in four main categories

  • page Load waitings: the first waiting to manage while testing your app, waiting for an event that allows you to understand that the page is interactive
  • content waitings: waiting for DOM element that matches a selector
  • XHR request waitings: waiting for an XHR request start or the corresponding response received
  • custom waitings: waiting for everything strictly related to the app that does not fall into the above categories

Every UI testing tool manages waitings in different ways, sometimes
automatically and sometimes manually. Below you can find some examples of
implementing the listed waitings.

Page load waitings

Every E2E testing tool manages the page load waiting in a different way (in terms of what is waited before considering the page loaded).

Cypress

cy.visit('http://localhost:3000')
Enter fullscreen mode Exit fullscreen mode

Puppeteer (and Playwright)

await page.goto('http://localhost:3000')
Enter fullscreen mode Exit fullscreen mode

Selenium

driver.get('http://localhost:3000')
driver.wait(function() {
  return driver
    .executeScript('return document.readyState')
    .then(function(readyState) {
      return readyState === 'complete'
    })
})
Enter fullscreen mode Exit fullscreen mode

TestCafé

fixture`Page load`.page`http://localhost:3000`
Enter fullscreen mode Exit fullscreen mode

Content waitings

Take a look at the following examples to see how waiting for a DOM element could be implemented in the available tools.

Cypress

  • waiting for an element: 
// it waits up to 4 seconds by default
cy.get('#form-feedback')
// the timeout can be customized
cy.get('#form-feedback', {timeout: 5000})
Enter fullscreen mode Exit fullscreen mode
  • waiting for an element with specific content 
cy.get('#form-feedback').contains('Success')
Enter fullscreen mode Exit fullscreen mode

Puppeteer (and Playwright)

  • waiting for an element: 
// it waits up to 30 seconds by default
await page.waitForSelector('#form-feedback')
// the timeout can be customized
await page.waitForSelector('#form-feedback', {timeout: 5000})
Enter fullscreen mode Exit fullscreen mode
  • waiting for an element with specific content 
await page.waitForFunction(
  selector => {
    const el = document.querySelector(selector)
    return el && el.innerText === 'Success'
  },
  {},
  '#form-feedback',
)
Enter fullscreen mode Exit fullscreen mode

Selenium 2

  • waiting for an element: 
driver.wait(until.elementLocated(By.id('#form-feedback')), 4000)
Enter fullscreen mode Exit fullscreen mode
  • waiting for an element with specific content 
const el = driver.wait(until.elementLocated(By.id('#form-feedback')), 4000)
wait.until(ExpectedConditions.textToBePresentInElement(el, 'Success'))
Enter fullscreen mode Exit fullscreen mode

TestCafé 2

  • waiting for an element: 
// it waits up to 10 seconds by default
await Selector('#form-feedback')
// the timeout can be customized
await Selector('#form-feedback').with({timeout: 4000})
Enter fullscreen mode Exit fullscreen mode
  • waiting for an element with specific content 
await Selector('#form-feedback').withText('Success')
Enter fullscreen mode Exit fullscreen mode

DOM Testing Library 1

  • waiting for an element: 
await findByTestId(document.body, 'form-feedback')
Enter fullscreen mode Exit fullscreen mode
  • waiting for an element with specific content 
const container = await findByTestId(document.body, 'form-feedback');
await findByText(container, 'Success')
Enter fullscreen mode Exit fullscreen mode

XHR request waitings

Cypress

  • waiting for an XHR request/response 
cy.intercept('http://dummy.restapiexample.com/api/v1/employees').as('employees')
cy.wait('@employees')
  .its('response.body')
  .then(body => {
    /* ... */
  })
Enter fullscreen mode Exit fullscreen mode

Puppeteer (and Playwright)

  • waiting for an XHR request 
await page.waitForRequest('http://dummy.restapiexample.com/api/v1/employees')
Enter fullscreen mode Exit fullscreen mode
  • waiting for an XHR response 
const response = await page.waitForResponse(
  'http://dummy.restapiexample.com/api/v1/employees',
)
const body = response.json()
Enter fullscreen mode Exit fullscreen mode



Even if it's a really important point, at the time of writing (July, 2019) it seems that waiting for XHR requests and responses is not so common. With exceptions for Cypress and Puppeteer, other tools/frameworks force you to look for something in the DOM that reflects the XHR result instead of looking for the XHR request itself. You can read more about the topic:

Custom waitings

The various UI testing tools/frameworks have built-in solutions to perform a lot of checks, but let's concentrate on writing a custom waiting. Since UI testing is 100% asynchronous, a custom waiting should face recursive promises, a concept not so handy to manage at the beginning.

Luckily, there are some handy solutions and plugins to help us with that.
Consider if we want to wait until a global variable (foo) is assigned with a particular value (bar): below you are going to find some examples.

Code Examples

Cypress

Thanks to the cypress-wait-until plugin you can do:

cy.waitUntil(() => cy.window().then(win => win.foo === 'bar'))
Enter fullscreen mode Exit fullscreen mode

Puppeteer (and Playwright)

await page.waitForFunction('window.foo === "bar"')
Enter fullscreen mode Exit fullscreen mode

Selenium 2

browser.executeAsyncScript(`
  window.setTimeout(function(){
    if(window.foo === "bar") {
      arguments[arguments.length - 1]();
    }
  }, 300);
`)
Enter fullscreen mode Exit fullscreen mode

TestCafé 2

const waiting = ClientFunction(() => window.foo === 'bar')
await t.expect(waiting()).ok({timeout: 5000})
Enter fullscreen mode Exit fullscreen mode

DOM Testing Library 1

await wait(() => global.foo === 'bar')
Enter fullscreen mode Exit fullscreen mode



1: unlike Cypress, Puppeteer, etc. DOM Testing Library is quite a different tool, that's why the examples are not available for every single part.


2: if there are better solutions or plugins to do the same, please let me know! I know Cypress, Puppeteer, and DOM Testing Library pretty well, but I can not say the same for Selenium and TestCafé.

Top comments (2)

Collapse
 
ramospedro profile image
Pedro •

Great tips!

I've seen this kind of "fixed wait" in practice and it's not pretty.

Before we know it, we're facing a magical 15 seconds wait in a lot of different places.
This makes the test suite take more time than it should, thus leading to drag and wasting money on CI machines.

Collapse
 
noriste profile image
Stefano Magni •

Sorry to hear about that, you end up hating E2E testing because of some bad practices 😓
BTW: thank you 😊

Read next

ronika_kashyap profile image

5 Best Mobile App Testing Tools in Today’s Dynamic Landscape

Ronika Kashyap -

jacobandrewsky profile image

Opinionated Nuxt Starter

Jakub Andrzejewski -

testifytech profile image

Differences Between Bug Severity and Priority in Testing [with Examples]

Steve Wortham -

codesolutionshub profile image

Common Causes of Slow Website Load Times and How to Fix Them 🚀

CodeSolutionsHub -