Netlify specialises in build tooling and hosting for JAMstack sites. A fundamental attribute of JAMstack sites is that they are pre-rendered and can be served directly from a CDN without the need to maintain logic on an origin web server.
Aside from the advantages relating to performance, security, scalability, and developer experience, this attribute unlocks some interesting capabilities.
One such ability is delivering A/B or multi-variant testing without the need for client-side UI manipulation or other JavaScript intervention. Doing this without client-side JavaScript helps to avoid impacting the site’s performance, and therefore the user experience, and the results of testing different UI or content.
In this post, I’ll explain how this works and how you can set up split testing (as we call it) on Netlify for a Gatsby site.
This approach works for all sites deployed to, and hosted on Netlify regardless of which static site generator or build tool used to create it. In the example video below, I’ll use Gatsby, but could equally have used Jekyll, Hugo, Eleventy, or any tools used to generate a site.
Creating different variants
Powering this feature are 2 underlying capabilities on Netlify: Branch Deploys and Edge Logic on Netlify’s ADN.
Branch Deploys
Netlify’s CI/CD pipeline builds on established git actions. Each push to the master branch (or an otherwise specified production branch) in your git repo will trigger a new build and deployment. This ensures that your version control reaches all the way to your production hosting environment.
Very cool.
However you can also tell Netlify to build and deploy changes pushed to any other branches in your git repository. When that happens, Netlify will automatically create another deployment target with a URL based on the name of your branch. Such deploys are identical to the deploys to your master branch in terms of infrastructure and hosting performance, so they are excellent candidates for testing and QA.
If you’ve ever worked on a traditional stack and sunk days (or perhaps weeks or even months!) of effort into creating identical, multiple environments for testing, staging, QA, production, etc… this should come as a welcome relief. With Netlify, you can create unlimited environments instantly, and they all behave exactly as the production environment because they all live together on the same underlying infrastructure. And if you like to work with feature branches, this approach gives you a genuine testable URL for every feature you create on its own feature branch.
With this feature in place, we can use it to power a split testing capability. Just as long as we can silently shape traffic to different branches at the CDN level.
Enter the edge logic of Netlify’s Application Delivery Network (ADN).
Edge Logic
Netlify serves all sites directly from a distributed CDN. But Netlify have highly optimised and enriched this CDN layer to give it some superpowers which you can control with disarming ease.
This enriched CDN is what we call the ADN or Netlify Edge.
One such superpower is to silently direct your site visitors to any of your site’s branches without them needing to know about the underlying branch URL. This can be configured in the Netlify admin under the Split Testing option.
Here you can nominate as many branches as you like and determine what percentage of site traffic will be directed to each of these branches. Once you’ve chosen how you’d like the traffic to be split, a single click to the Start Test button will tell Netlify’s ADN to begin shaping the incoming traffic according to those weightings. Associating a visitor with a particular branch happens out at the ADN and includes the silent assignment of a cookie which ensures that each user has affinity with a branch. This keeps their experiences consistent while a test is running. In the video you’ll see that I break out Incognito mode and another browser until I get routed to each of my test branches.
Making the difference
There are already a number of services out there which provide split testing capabilities, so what sets this apart?
Performance.
And as we often hear. Performance matters.
The most common approach to split testing, or multi-variant testing used client side JavaScript to make changes to the UI for different users. The performance overheads of this can be significant since for this to work, the following must take place:
- Site requested
- Site delivered from host
- JavaScript requested from third party split testing service
- JavaScript returned
- JavaScript parsed
- JavaScript executed
- Additional assets requested if required (by JavaScript)
- Additional assets returned
- DOM manipulated with updated UI
This introduces a lot of activity which all takes time. If everything is successful we’ll still be introducing a delay before the interface is updated and ready to use.
Given that performance is a key contributor to user experience and conversion, this kind of delay will significantly skew the results. What we want is to be able to test like-for-like performance without adding any other factors which could taint the results.
When Netlify’s Split Testing direct users to one branch or another, it does so without adding any additional requests or altering the performance profile of the infrastructure in any way. On Netlify, all branches are production branches, and so our users are seeing the nominated variant of the user interface as a result of the following flow:
- Site requested
- Site delivered from host
This means our tests are testing the actual differences in our sites, and not any of the artefacts of the testing tools.
In the video above, you’ll see how I set this up to enable A/B testing for a Gatsby site. In doing this, I’m also adding branch deploys, a continuous delivery pipeline, and automated builds which generate immutable deployment. It takes just a few minutes, but includes a surprising number of powerful features offered by default on the Netlify platform. Underlying platform features like this take the experience of developing and serving a JAMstack site from good, to great. No matter what technology you chose to build your site with.
Try it for yourself
The link below will clone this same example repository to your own GitHub account and initiate a new site deploy in Netlify. My video showed the entire process in under 10 minutes. I’d be willing to bet you could set it all up in less.
Ready?
Go! (Clone the example Gatsby site as a new site on Netlify)
Top comments (0)