DEV Community

Ryan Feigenbaum for GrowthBook

Posted on • Originally published at blog.growthbook.io on

Changing Running Experiments Safely and Flexibly in GrowthBook

Changing Running Experiments Safely and Flexibly in GrowthBook

Running experiments can be a messy business and you often want to make changes mid-experiment.

Making changes to running experiments in GrowthBook 2.7 is:

  • Safer than ever. With guided flows that ensure you don’t introduce bias when changing targeting or traffic rules, you are able to pick the least disruptive deployment strategy for your changes.
  • More flexible than ever. Sticky bucketing allows you to preserve user experience when making certain kinds of changes.

This articles walks you through three kinds of changes you can make in GrowthBook, how our UI helps you navigate them, and how sticky bucketing can help you make changes safely.

Decreasing traffic to an experiment

In some cases, you may wish to decrease traffic to an experiment, either because you have enough users and want to ramp down new enrollment, or you want to begin restricting your experiment to some subset of users using more restrictive targeting attributes.

Imagine you add a targeting attribute to only target US-based users for your experiment. If you simply make the change and push it live, then your experiment analysis will include users from before you pushed the change and they are no longer eligible for the experiment. They will still be in your experimental sample which means their behavior may be affecting your experiment results, but they’ll no longer be receiving the same feature values as before since they are now excluded by the new targeting rules.

In this case, you have three options:

  • New phase, re-randomize users. This ensures your analysis is accurate, but it throws away your existing data and it may change users’ experiences.
  • Same phase, apply changes to everyone. This lets you leverage existing data with the knowledge that some users may be in your experimental data but are no longer receiving the new experiment feature, therefore biasing your results.
  • In GrowthBook 2.7, we have added sticky bucketing, which, if enabled for your org, provides you a third option: Same Phase, apply changes to new traffic only. This lets you (a) keep all of your data in your analysis, (b) ensure users do not change their originally assigned variation, and (c) avoid suffering any bias as users in each variation will continue receiving those feature values.

Here’s a loom showing the flow in action

Increasing traffic to an experiment

Increasing traffic to an experiment is less problematic. You can safely make any of the following changes without starting a new phase:

  • Increasing the percent of traffic to an experiment
  • Removing restrictive targeting (i.e. making targeting more permissive)
  • Removing an experiment from a Namespace

Changing Running Experiments Safely and Flexibly in GrowthBook

Restarting an experiment, or starting a new phase

Sometimes, we need to start an experiment over and throw away our old data, either due to a bug in implementation, a change in experiment design, or if we want to restart an A/A test to make sure that some imbalance was due to chance and not due to an issue with your GrowthBook implementation.

In these cases, starting a new phase of the experiment will require that you re-randomize in order to avoid carry over bias. You can read more about carryover bias in our docs.

As with other changes, we provide this information directly in the app to ensure you can make fully informed choices.

Changing Running Experiments Safely and Flexibly in GrowthBook

Feel free to check our our docs on sticky bucketing or making changes to running experiments for more detail.

Image of AssemblyAI tool

Challenge Submission: SpeechCraft - AI-Powered Speech Analysis for Better Communication

SpeechCraft is an advanced real-time speech analytics platform that transforms spoken words into actionable insights. Using cutting-edge AI technology from AssemblyAI, it provides instant transcription while analyzing multiple dimensions of speech performance.

Read full post

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay