A/B Testing with the new Next.js 12 Middleware

Vercel recently released Next.js 12 which adds a number of exciting performance improvements as well as a new beta feature - Middleware. Middleware has many uses, but I'm going to focus in this post on A/B Testing.

You've always been able to run A/B tests on Next.js applications, but until this latest release there have been some major strings attached. For example, on static pages, there would often be a "flash" where users would see the original page for a split second before your variation popped in and replaced it. And on server-rendered pages, you would need to completely disable caching since two users on the same URL could get two different HTML responses.

Next.js middleware fixes these issues in an elegant way. You can create two different versions of a page and using a single URL, route traffic between them with a middleware function. The middleware is run at the edge, so it's globally distributed and super fast for your users.

Setting up the Next.js App

We'll start with a standard Typescript Next.js app:

npx create-next-app@latest --ts
cd my-app
npm run dev

Now you should be able to visit http://localhost:3000 and see a homepage.

Let's create a new version of the homepage at pages/new_homepage.tsx:

export default function NewHomePage() {
  return <h1>Welcome to my new site!</h1>
}

Now you should have two working URLs:

1. The original homepage - http://localhost:3000
2. The new (way better) homepage - http://localhost:3000/new_homepage

Our goal is instead of having these on two separate URLs, we want 50% of the visitors to the root URL (/) to see the original page and the other 50% to see the new one.

That sounds like an A/B test! To help do the traffic splitting, we're going to use GrowthBook, an open source feature flagging platform with really robust A/B testing support.

Setting up GrowthBook

You can self-host GrowthBook (https://github.com/growthbook/growthbook) or create a free Cloud account (https://app.growthbook.io/). Either way, once you login, there are a couple steps required before you can run an experiment.

First, click on Step 1: Install our SDK and run the npm install command:

npm i --save @growthbook/growthbook

Note: Next.js middleware runs outside of a React context, so we're using the vanilla Javascript SDK above instead of the React one.

Creating the Next.js Middleware

Now, we'll integrate the sample code in GrowthBook into our Next.js App. Create a file pages/_middleware.ts with the following contents (make sure to swap out the placeholder with the unique API endpoint you see in GrowthBook):

import { NextRequest, NextResponse } from 'next/server'
import { GrowthBook } from '@growthbook/growthbook'

const FEATURES_ENDPOINT = 'YOUR_GROWTHBOOK_ENDPOINT_HERE'

// Fetch features from GrowthBook API and cache in memory
let features = null;
let lastFetch = 0;
async function getFeatures() {
  if (Date.now() - lastFetch > 5000) {
    lastFetch = Date.now();
    const latest = fetch(FEATURES_ENDPOINT)
      .then(res => res.json())
      .then(json => features = json.features || features)
      .catch((e) => console.error("Error fetching features", e))
    // If this is the first time, wait for the initial fetch
    if(!features) await latest;
  }
  return features || {};
}

const COOKIE = 'visitor_id'

export async function middleware(req: NextRequest) {
  // We only want to run the A/B test on the homepage
  const pathname = req.nextUrl.pathname;
  if (pathname !== "/") {
    return NextResponse.next()
  }

  // Get existing visitor cookie or create a new one
  let visitor_id = req.cookies[COOKIE] || crypto.randomUUID()

  // Create a GrowthBook client instance
  const growthbook = new GrowthBook({
    attributes: { id: visitor_id },
    features: await getFeatures(),
    trackingCallback: (exp, res) => {
      console.log("In Experiment", exp.key, res.variationId);
    }
  });

  // Pick which page to render depending on a feature flag
  let res = NextResponse.next();
  if (growthbook.feature("new-homepage").on) {
    const url = req.nextUrl.clone();
    url.pathname = "/new_homepage";
    res = NextResponse.rewrite(url);
  }

  // Store the visitor cookie if not already there
  if (!req.cookies[COOKIE]) {
    res.cookie(COOKIE, visitor_id)
  }

  return res
}

There's a lot going on here, but it's not too hard to follow:

1. Function to fetch feature definitions from the GrowthBook API, cache them, and keep it up-to-date
2. Skip the middleware if the user is requesting any page other than /
3. Look for an existing visitor id stored in a cookie and generate one if it doesn't exist yet.
4. Create a GrowthBook client instance
5. Determine which page to render based on a GrowthBook feature flag
6. Set the visitor id cookie on the response if needed
7. Return the response

Creating the Feature Flag

At this point, if you visit http://localhost:3000 you'll always see the original homepage still.

This is because the code is looking for a feature flag named new-homepage, which doesn't exist yet. Flags that don't exist yet are always treated as if they are off so the middleware just returns the original page. Let's fix this.

Back in GrowthBook, close the SDK instructions modal and click on Step 2: Add your first feature. Enter the feature key new-homepage. Keep the feature type set to on/off, choose "A/B Experiment" as the behavior, and leave everything else set to the default (split users by id, even 50/50 split, "new-homepage" as the tracking key).

Click save, wait a few seconds, and refresh your NextJS app.

Depending on the random visitor_id cookie that the middleware generated, you may see either version of the homepage. You can delete that cookie and refresh a few times. You'll notice about half the time you get the new page and the other half you don't.

Also, if you look in the terminal where you're running the Next.js npm run dev command, you should see the log messages from trackingCallback.

Analyzing Results

Just running an A/B test by itself is fun, but not that useful. You also need to track and analyze the results.

In the trackingCallback in pages/_middleware.ts, instead of doing a console.log, we could use Mixpanel or Segment or another event tracking system.

Then, in the app, we could similarly track events when the users do something we care about, like sign up or buy something.

Once you do that, GrowthBook can connect to your event tracking system, query the raw data, run it through a stats engine, and show you the results. This process is a little more involved to set up, but I will walk through it in a followup post.

The Future

Hopefully in future releases, Next.js expands on their middleware feature to make A/B testing even more powerful. Imagine, for example, that middleware could inject props into your pages, similar to getServerSideProps. Then you wouldn't need to create new temporary pages every time you wanted to run an A/B test!

Comments

[Amantel]

Hi!
Great guide, thanks.

This line

  if (growthbook.feature("new-homepage").on) {

Probably is better written as

if (growthbook.feature("new-homepage").isOn) {

Original code work, but it's kinda confusing that sdk docs differ from the guide.

[Jeremy Dorn]

isOn is just a helper method and is used like this to make the code a little shorter:

if (growthbook.isOn("new-homepage")) {

[Afzal Hossain]

Worked on first try but with one issue. Can't maintain sticky session when using with netlify. visitor_id/id is not sent to growthbook and it's kept locally and when we have multiple instance if the middleware one isn't aware of the other and the A/B testing version initially served don't stick and randomly switch to different version for the same user (assuming when the middleware instance changes).
Thinking about workarounds (e.g separate cookie for each feature) but then I probably can't use most of the SDK features anymore :(
Anyone has a nice solution to this issue?

[Collin Caram]

How can we keep the middleware fast and avoid making a call to the growthbook endpoint? Any way we can have that information cached locally?

[Jeremy Dorn]

I don't believe Next.js provides any sort of persistent storage or cache for middleware yet. But, I do think you can do some in-memory caching. Basically, if you do the fetch outside of the exported middleware function and store the response in a variable, it should only run once at startup and be re-used across invocations. I haven't tested this with Vercel, but that's how Cloudflare workers and Lambda does it at least.

I can do some tests and update the article if works and ends up being better for performance.

[Collin Caram]

Awesome, yeah the problems that would be nice if solved are:

1. how can we maintain optimal performance while doing a/b testing?
2. how can we update a/b testing config in growthbook, and have that be reflected in the application?

[Jeremy Dorn]

@collinc777 I updated the example code in the article to use in-memory caching. So now, the first request will wait for the API response, but subsequent ones will return immediately. And it will refresh in the background after 5 seconds to keep the features up-to-date.

Still some improvements to make this truly production ready, like setting a time-out for the API fetch, but it's much closer now. Thanks for the idea!

[Collin Caram]

Dang that was fast. Thanks @jdorn! I appreciate you being so responsive!

[valentin mundrov]

I understand the Next 12.2.0 middleware runs on edge-runtime and won't let us fetch data without proxy (Prisma). Edge-runtime requires req to be passed in <1.5sec. or it times out. So, is fetching in middleware actually possible? Appreciate your effort!

[Prajjwal]

Cool! and so quick post!!

[Ludal 🚀]

Wow, I didn't even know this could be possible with Next.js! I'm so impressed by their updates. Thanks for sharing. 🤝

[david ornelas]

Do you have a repo somewhere? I cant seem to get growthbook to work

[Jeremy Dorn]

Hi David. What problems are you having?

I just found and fixed a couple typos in the middleware code in the article. It was using /homepage_new instead of /new_homepage and I forgot an async in the function signature. It's possible those were causing issues for you.

[david ornelas]

I have a cloud account on growthbook.io and it cant seem to connect to it to check whether to roll out a feature. Everything just says "unknownsource" on it

[Christopher N. KATOYI]

This is huge. Thanks for sharing

[Anish De]

Great article, keep it up!!!

[KobeTwo]

Is it neccessary to block the variant pages from direct access so that a customer is not able to access them via URL?