Measuring the impact of a test variation in React with Amplitude and ConfigCat

To keep an app or website functioning, a business may choose to roll out new features or make updates. If these updates or features are somehow shipped with bugs and are not well received by users, this may cause a decline in user engagement and can lead to revenue losses.

To prevent these sorts of consequences a business can determine how much of an impact (Positive or Negative) a new change/feature could have by performing an A/B test to measure the impact before permanently making a change. For example, They can roll out a new feature/ update to a subset of users and monitor the impact before deciding to reach more users.

Implementing A/B tests can be streamlined with the help of feature flags.

What are Feature Flags

If you have never heard of feature flags before, You can think of them as switches or feature toggles that allow you to turn a feature on or off in your website or application. With this control, you can easily roll out a new feature in your app because you can turn that feature on and observe feedback from your users. Should anything go wrong, you can simply toggle the feature off.

For conciseness reasons, I’m going to assume that you’ve already integrated ConfigCat’s Feature Flag service in your app. If not, then the official docs have you covered.

How can feature flags simplify the A/B testing experience?

When it comes to A/B Testing, feature flags give us the ability to toggle a new change/ variant on or off based on certain demographics or user groups, so we can gather performance data on the change before deciding whether the change should go onto production or not.

For example, the general idea behind my example would involve creating a feature flag called isShowDollarAmountDiscountEnabled in the ConfigCat Dashboard that you can toggle on or off right from the platform.

Let us explore the A/B test case

My objective is to determine if clicks on the Shop Now button would increase If I change the discount amount from a percentage(20% off) to a fixed value ($10 off).

What I currently have:

Get 20% OFF! When you shop today!

What I want to test:

Get $10 OFF! When you shop today!

Getting started

Here is the source code to the sample React app on GitHub.

To track the number of clicks on the Shop Now button, I will use an online platform called Amplitude. However, you must first create an account before you can use Amplitude. Here are the steps for doing so:

Setting up an Amplitude account

1. Go to https://amplitude.com/get-started

2. Complete the account setup. Since my sample app uses React, I’ll choose the JavaScript SDK. Do the same if you’re following along.

3. Run the following command in the root folder of your React app or website to install the amplitude package:

npm install amplitude-js

4. Let's Log an Event from the website.

a. Add the following line to the top of the App.js file:

import amplitude from 'amplitude-js';

b. Inside the App() function in App.js, add the following:

Note: API_KEY should be replaced with your actual Amplitude API key

const AmplitudeInstance = amplitude.getInstance().init("API_KEY");

IMPORTANT: If you're using a feature flag to toggle the variation, then you’d only want to log the Shop Now event if that feature is enabled.
c. Add a handleClick function that gets triggered when the Shop Now button is clicked:

  const handleClick = () => {
    // Log clicks to Amplitude
    amplitude.getInstance(AmplitudeInstance).logEvent("Shop Now");
  }

Notice below that I'm logging an event called Shop Now to Amplitude. Feel free to choose any name if you're following along.

(logEvent("Shop Now"))

d. Start the app and click "Show Now" to log the event to Amplitude.

Here is what the code looks like in App.js:

// Amplitude JS
import amplitude from 'amplitude-js';
import CartPlusIcon from "./components/CartPlusIcon";
function App() {
  // Initialize Amplitude
  const AmplitudeInstance = amplitude.getInstance().init("71b8a9d6b70f4f483351c4c200f5f1f4");
  const DiscountAmount = "20%"
  const handleClick = () => {
    // Log clicks to Amplitude
    amplitude.getInstance(AmplitudeInstance).logEvent("Shop Now");
  }
  return (
    <Container className="custom-jumbotron py-5 text-center">
      <Row className="py-lg-5">
        <Col lg={6} md={8} className="mx-auto">
          <h1 className="fw-light">Limited Time Discount</h1>
          <p className="subtext lead text-muted">Get <span className="fw-bold">{DiscountAmount}</span> OFF! when you shop today!</p>
          <Button onClick={handleClick} variant="dark">Shop Now <CartPlusIcon/></Button>
        </Col>
      </Row>
    </Container>
  );
}
export default App;

The ConfigCat platform can also directly integrate with Amplitude. Click here to read more.

Now that Amplitude has received the event, it's best to also set up an analysis chart to also display received click event details.

Setting up an analysis in Amplitude for Analyzing the button click event.

Here is how to set up an analysis chart:

1. Navigate to the workspace page on Amplitude and click the New button in the center:

2. Then select "Chart" from the top left:

3. Next, click the "Select event" dropdown and select the "Shop Now" event (the one that you registered earlier during the setup).

4. You can also add a description or even categorize the event if you want.

5. Click "Save" on the top right, then enter a name and description for your chart.

Quick Recap - What was done so far?

• I set up an Amplitude account
• Configured my React app to connect to the account using my Amplitude API Key
• Logged an event called Shop Now from React to Amplitude
• Started an analysis by creating a chart to track clicks on the Shop Now button.

Code

Feel free to check out completed code for this app.

Final thoughts

If this app was out on production, we could’ve compared how much of an impact such a change would have across thousands of users. The thing about A/B tests is that your users are able to help you determine whether you keep or remove features in your app or website by effectively “voting” with their user engagement.

It's worth noting that ConfigCat’s feature flag services help facilitate A/B testing, making the entire testing experience more seamless.

You can find ConfigCat on Twitter, Facebook, LinkedIn, and GitHub.