DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’» is a community of 966,904 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Create account Log in
Jeremy Dorn
Jeremy Dorn

Posted on • Updated on

A/B Testing with the new Next.js 12 Middleware

Vercel recently released Next.js 12 which adds a number of exciting performance improvements as well as a new beta feature - Middleware. Middleware has many uses, but I'm going to focus in this post on A/B Testing.

You've always been able to run A/B tests on Next.js applications, but until this latest release there have been some major strings attached. For example, on static pages, there would often be a "flash" where users would see the original page for a split second before your variation popped in and replaced it. And on server-rendered pages, you would need to completely disable caching since two users on the same URL could get two different HTML responses.

Next.js middleware fixes these issues in an elegant way. You can create two different versions of a page and using a single URL, route traffic between them with a middleware function. The middleware is run at the edge, so it's globally distributed and super fast for your users.

Setting up the Next.js App

We'll start with a standard Typescript Next.js app:

npx create-next-app@latest --ts
cd my-app
npm run dev
Enter fullscreen mode Exit fullscreen mode

Now you should be able to visit http://localhost:3000 and see a homepage.

Let's create a new version of the homepage at pages/new_homepage.tsx:

export default function NewHomePage() {
  return <h1>Welcome to my new site!</h1>
}
Enter fullscreen mode Exit fullscreen mode

Now you should have two working URLs:

  1. The original homepage - http://localhost:3000
  2. The new (way better) homepage - http://localhost:3000/new_homepage

Our goal is instead of having these on two separate URLs, we want 50% of the visitors to the root URL (/) to see the original page and the other 50% to see the new one.

That sounds like an A/B test! To help do the traffic splitting, we're going to use GrowthBook, an open source feature flagging platform with really robust A/B testing support.

Setting up GrowthBook

You can self-host GrowthBook (https://github.com/growthbook/growthbook) or create a free Cloud account (https://app.growthbook.io/). Either way, once you login, there are a couple steps required before you can run an experiment.

First, click on Step 1: Install our SDK and run the npm install command:

npm i --save @growthbook/growthbook
Enter fullscreen mode Exit fullscreen mode

Note: Next.js middleware runs outside of a React context, so we're using the vanilla Javascript SDK above instead of the React one.

Creating the Next.js Middleware

Now, we'll integrate the sample code in GrowthBook into our Next.js App. Create a file pages/_middleware.ts with the following contents (make sure to swap out the placeholder with the unique API endpoint you see in GrowthBook):

import { NextRequest, NextResponse } from 'next/server'
import { GrowthBook } from '@growthbook/growthbook'

const FEATURES_ENDPOINT = 'YOUR_GROWTHBOOK_ENDPOINT_HERE'

// Fetch features from GrowthBook API and cache in memory
let features = null;
let lastFetch = 0;
async function getFeatures() {
  if (Date.now() - lastFetch > 5000) {
    lastFetch = Date.now();
    const latest = fetch(FEATURES_ENDPOINT)
      .then(res => res.json())
      .then(json => features = json.features || features)
      .catch((e) => console.error("Error fetching features", e))
    // If this is the first time, wait for the initial fetch
    if(!features) await latest;
  }
  return features || {};
}

const COOKIE = 'visitor_id'

export async function middleware(req: NextRequest) {
  // We only want to run the A/B test on the homepage
  const pathname = req.nextUrl.pathname;
  if (pathname !== "/") {
    return NextResponse.next()
  }

  // Get existing visitor cookie or create a new one
  let visitor_id = req.cookies[COOKIE] || crypto.randomUUID()

  // Create a GrowthBook client instance
  const growthbook = new GrowthBook({
    attributes: { id: visitor_id },
    features: await getFeatures(),
    trackingCallback: (exp, res) => {
      console.log("In Experiment", exp.key, res.variationId);
    }
  });

  // Pick which page to render depending on a feature flag
  let res = NextResponse.next();
  if (growthbook.feature("new-homepage").on) {
    const url = req.nextUrl.clone();
    url.pathname = "/new_homepage";
    res = NextResponse.rewrite(url);
  }

  // Store the visitor cookie if not already there
  if (!req.cookies[COOKIE]) {
    res.cookie(COOKIE, visitor_id)
  }

  return res
}
Enter fullscreen mode Exit fullscreen mode

There's a lot going on here, but it's not too hard to follow:

  1. Function to fetch feature definitions from the GrowthBook API, cache them, and keep it up-to-date
  2. Skip the middleware if the user is requesting any page other than /
  3. Look for an existing visitor id stored in a cookie and generate one if it doesn't exist yet.
  4. Create a GrowthBook client instance
  5. Determine which page to render based on a GrowthBook feature flag
  6. Set the visitor id cookie on the response if needed
  7. Return the response

Creating the Feature Flag

At this point, if you visit http://localhost:3000 you'll always see the original homepage still.

This is because the code is looking for a feature flag named new-homepage, which doesn't exist yet. Flags that don't exist yet are always treated as if they are off so the middleware just returns the original page. Let's fix this.

Back in GrowthBook, close the SDK instructions modal and click on Step 2: Add your first feature. Enter the feature key new-homepage. Keep the feature type set to on/off, choose "A/B Experiment" as the behavior, and leave everything else set to the default (split users by id, even 50/50 split, "new-homepage" as the tracking key).

Click save, wait a few seconds, and refresh your NextJS app.

Depending on the random visitor_id cookie that the middleware generated, you may see either version of the homepage. You can delete that cookie and refresh a few times. You'll notice about half the time you get the new page and the other half you don't.

Also, if you look in the terminal where you're running the Next.js npm run dev command, you should see the log messages from trackingCallback.

Analyzing Results

Just running an A/B test by itself is fun, but not that useful. You also need to track and analyze the results.

In the trackingCallback in pages/_middleware.ts, instead of doing a console.log, we could use Mixpanel or Segment or another event tracking system.

Then, in the app, we could similarly track events when the users do something we care about, like sign up or buy something.

Once you do that, GrowthBook can connect to your event tracking system, query the raw data, run it through a stats engine, and show you the results. This process is a little more involved to set up, but I will walk through it in a followup post.

The Future

Hopefully in future releases, Next.js expands on their middleware feature to make A/B testing even more powerful. Imagine, for example, that middleware could inject props into your pages, similar to getServerSideProps. Then you wouldn't need to create new temporary pages every time you wanted to run an A/B test!

Top comments (17)

Collapse
 
amantel profile image
Amantel • Edited on

Hi!
Great guide, thanks.

This line

  if (growthbook.feature("new-homepage").on) {
Enter fullscreen mode Exit fullscreen mode

Probably is better written as

if (growthbook.feature("new-homepage").isOn) {
Enter fullscreen mode Exit fullscreen mode

Original code work, but it's kinda confusing that sdk docs differ from the guide.

Collapse
 
jdorn profile image
Jeremy Dorn Author

isOn is just a helper method and is used like this to make the code a little shorter:

if (growthbook.isOn("new-homepage")) {
Enter fullscreen mode Exit fullscreen mode
Collapse
 
afzalh profile image
Afzal Hossain

Worked on first try but with one issue. Can't maintain sticky session when using with netlify. visitor_id/id is not sent to growthbook and it's kept locally and when we have multiple instance if the middleware one isn't aware of the other and the A/B testing version initially served don't stick and randomly switch to different version for the same user (assuming when the middleware instance changes).
Thinking about workarounds (e.g separate cookie for each feature) but then I probably can't use most of the SDK features anymore :(
Anyone has a nice solution to this issue?

Collapse
 
collinc777 profile image
Collin Caram

How can we keep the middleware fast and avoid making a call to the growthbook endpoint? Any way we can have that information cached locally?

Collapse
 
jdorn profile image
Jeremy Dorn Author

I don't believe Next.js provides any sort of persistent storage or cache for middleware yet. But, I do think you can do some in-memory caching. Basically, if you do the fetch outside of the exported middleware function and store the response in a variable, it should only run once at startup and be re-used across invocations. I haven't tested this with Vercel, but that's how Cloudflare workers and Lambda does it at least.

I can do some tests and update the article if works and ends up being better for performance.

Collapse
 
collinc777 profile image
Collin Caram

Awesome, yeah the problems that would be nice if solved are:

  1. how can we maintain optimal performance while doing a/b testing?
  2. how can we update a/b testing config in growthbook, and have that be reflected in the application?
Thread Thread
 
jdorn profile image
Jeremy Dorn Author

@collinc777 I updated the example code in the article to use in-memory caching. So now, the first request will wait for the API response, but subsequent ones will return immediately. And it will refresh in the background after 5 seconds to keep the features up-to-date.

Still some improvements to make this truly production ready, like setting a time-out for the API fetch, but it's much closer now. Thanks for the idea!

Thread Thread
 
collinc777 profile image
Collin Caram

Dang that was fast. Thanks @jdorn! I appreciate you being so responsive!

Collapse
 
valento profile image
valentin mundrov

I understand the Next 12.2.0 middleware runs on edge-runtime and won't let us fetch data without proxy (Prisma). Edge-runtime requires req to be passed in <1.5sec. or it times out. So, is fetching in middleware actually possible? Appreciate your effort!

Collapse
 
monarch profile image
Prajjwal

Cool! and so quick post!!

Collapse
 
iamludal profile image
Ludal πŸš€ • Edited on

Wow, I didn't even know this could be possible with Next.js! I'm so impressed by their updates. Thanks for sharing. 🀝

Collapse
 
davidornelas11 profile image
david ornelas

Do you have a repo somewhere? I cant seem to get growthbook to work

Collapse
 
jdorn profile image
Jeremy Dorn Author

Hi David. What problems are you having?

I just found and fixed a couple typos in the middleware code in the article. It was using /homepage_new instead of /new_homepage and I forgot an async in the function signature. It's possible those were causing issues for you.

Collapse
 
davidornelas11 profile image
david ornelas • Edited on

I have a cloud account on growthbook.io and it cant seem to connect to it to check whether to roll out a feature. Everything just says "unknownsource" on it

Collapse
 
christopher2k profile image
Christopher N. KATOYI

This is huge. Thanks for sharing

Collapse
 
anishde12020 profile image
Anish De

Great article, keep it up!!!

Collapse
 
kobetwo profile image
KobeTwo

Is it neccessary to block the variant pages from direct access so that a customer is not able to access them via URL?

🌚 Friends don't let friends browse without dark mode.

Sorry, it's true.