Craig Morten

Posted on Apr 3, 2022 • Updated on Jan 15, 2023

Testing Web Vitals With Cypress

#javascript #webdev #testing #tutorial

It is well understood that performance is a critical consideration for any website which can have far reaching impacts on everything from customer satisfaction, SEO, and ultimately your bottom line. You cannot determine the success of performance work without the ability measure the results and compare to performance budgets - this calls for testing infrastructure to make sure you have the necessary visibility on metrics... introducing cypress-web-vitals.

cypress-web-vitals allows you to test against the Google Web Vital signals within your Cypress workflows through a new cy.vitals() custom command.

Web Vitals is an initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.

Reference: https://web.dev/vitals/

Getting started

In your project, install the dependencies:

npm install --save-dev cypress-web-vitals cypress-real-events

Note: cypress-web-vitals currently makes use of cypress-real-events to click the page to calculate first input delay. Hence it is needed as a peer-dependency.

Within you support commands file (normally cypress/support/commands.js), add this one liner to get setup:

import "cypress-web-vitals";

And now you're set to get going with Web Vital performance budget tests in your Cypress workflow! 🎉

Add you first test like so:

describe("Web Vitals", () => {
  it("should pass the meet Google's 'Good' thresholds", () => {
    cy.vitals({ url: "https://www.google.com/" });
  });
});

You are now set up to test against all of the Web Vitals using Google's "Good" threshold values:

Largest contentful paint (LCP) - 2500.
First input delay (FID) - 100.
Cumulative layout shift (CLS) - 0.1.
First contentful paint (FCP) - 1800.
Time to first byte (TTFB) - 600.

Customise your tests

You can further customise your tests through additional optional configuration which is passed to the cy.vitals(webVitalsConfig) call:

Optional url: string - The url to audit. If not provided you will need to have called cy.visit(url) prior to the command.
Optional firstInputSelector: string - The element to click for capturing FID. The first matching element is used. Default: "body".
Optional thresholds: object - The thresholds to audit the Web Vitals against. If not provided Google's "Good" thresholds will be used. If provided, any missing Web Vitals signals will not be audited.
Optional vitalsReportedTimeout: number - Time in ms to wait for Web Vitals to be reported before failing. Default: 10000.

For example:

// Use the `main` element for clicking to capture the FID.
cy.vitals({ firstInputSelector: "main" });

// Test the page against against a CLS threshold of 0.2.
cy.vitals({ thresholds: { cls: 0.2 } });

For more details on usage refer to the API docs.

How does it work?

The url is visited with the HTML response intercepted and modified by Cypress to include the web-vitals script and some code to record the Web Vitals values.
Several elements (if exist) starting with the provided element (based on firstInputSelector) are then clicked in quick succession to simulate a user clicking on the page. Note: if choosing a custom element, don't pick something that will navigate away from the page otherwise the plugin will fail to capture the Web Vitals metrics.
The audit then waits for the page load event to allow for the values of LCP and CLS to settle; which are subject to change as different parts of the page load into view.
Next the audit simulates a page visibility state change which is required for the CLS Web Vital to be reported.
The audit then attempts to wait for any outstanding Web Vitals to be reported for which thresholds have been provided.
Finally the Web Vitals values are compared against the thresholds, logging successful results and throwing an error for any unsuccessful signals. Note: if the audit was unable to record a Web Vital then it is logged, but the test will not fail.

Testing sites in the wild

Here are some example test runs against FAANG homepages to see cypress-web-vitals in action:

Facebook

cy.vitals({ url: "https://www.facebook.com/" });

Amazon

cy.vitals({ url: "https://www.amazon.com/" });

Netflix

cy.vitals({
  url: "https://www.netflix.com/gb/",
  firstInputSelector: ".our-story-card-title",
});

For Netflix we have had to introduce a custom element choice for the simulated "first click". Even though first input delay can be measured when in cases where there is no event listener registered to the element, there are scenarios where clicking the body doesn't cut it. Some good examples of elements that will reliably trigger an FID metric are:

Text fields, checkboxes, and radio buttons (<input>, <textarea>)

Select dropdowns (<select>)

Links (<a>)

In order to ensure the Web Vitals can be tested against, it is best to try find an element that reliably triggers an FID, but won't navigate away from the page (perhaps avoid <a>!).

Google

cy.vitals({ url: "https://www.google.com/" });

Wrap-up

Using any awesome performance testing tooling lately? Tried out cypress-web-vitals on your site and have results to share? Got any comments, queries, or questions? Leave a comment below!

That's all folks! 🚀

Top comments (6)

irène • May 1 '23

Hey @craigmorten thanks for the article!

Using cypress-web-vitals I noticed the results of a page are different compared to results I have running Lighthouse from the browser for the same page. Did this happen to you too? Thanks

Craig Morten • May 2 '23 • Edited

Hey @ir3ne 👋

Yes I would expect the results to be different. You will encounter some natural variation regardless (even between runs on the same page with lighthouse) due to volatility in network etc., but beyond that I’d generally expect the results to be slower reported by this package than by Lighthouse.

With this package we’re running the page instrumented with Cypress which will have a performance impact compared with just loading the page in the browser by itself. It also is injecting the web-vitals snippet into the page to get the results which suffers from the Observer Effect of introducing extra page weight (though minimised best as possible). Lighthouse (assuming devtools) doesn’t suffer from these points so likely gives a more accurate reading of performance (unless using lighthouse and Cypress, in which case it depends).

If you want your “lab” results to be as close to real user performance then lighthouse is likely better (though with the caveat that even lighthouse isn’t realistic, only real user monitoring can give a real gauge!). If using for relative performance budgeting/benchmarks then either is hopefully fine 🙃