DEV Community

Cover image for Why "works on my machine" often means "I'm still in variant B" — QA rules for testing behind A/B experiments and feature flags
Denis Skvortsov
Denis Skvortsov

Posted on

Why "works on my machine" often means "I'm still in variant B" — QA rules for testing behind A/B experiments and feature flags

#opensource #productivity #testing #showdev

If your product ships behind A/B experiments and feature flags, your QA team has a failure mode nobody tracks: variant amnesia. You add yourself to an experiment's testing group to check a feature, switch variants a couple of times, move on. A week later you're testing something else, the app behaves strangely, and you file a bug — not knowing you're still enrolled in variant B of an experiment you forgot exists.

I lead QE on a team that works with Amplitude daily. Here's what variant amnesia actually costs, the rules that fixed it, and the tool I built to make the rules cheap.

The cost

  • False bugs. The most expensive kind: a developer spends an hour reproducing an issue that is just an experiment variant doing its job.
  • Irreproducible reports. Two testers see different UI on the same build, compare screenshots, and argue — because one of them is enrolled and the other isn't. The repro steps are useless without enrollment state, and nobody writes enrollment state into bug reports.
  • Cleanup archaeology. In Amplitude, checking where you're enrolled means opening every experiment one by one and scanning its testing list for your email. Nobody does that weekly. So nobody knows their own state.

The rules that fixed it

  1. Test on aliases, never on your main account. you+checkout@company.com, you+paytest1@company.com — disposable identities with clean state, one per scenario. Your main account stays out of every testing group.
  2. Treat enrollment as part of the repro steps. A bug report on an experiment-covered surface states which identity was enrolled in what. If you can't say, you're not done investigating.
  3. Leave tests when you're done. Exiting the testing group is exit criteria of the test cycle, same as closing the ticket. Stale enrollments are the amnesia.
  4. Check enrollment first, debug second. When the app "behaves strangely," the first question is not "what broke" but "what am I enrolled in."

These rules work on any platform — LaunchDarkly, Optimizely, GrowthBook. The problem is that on all of them, following the rules by hand is tedious. Tedious rules get skipped.

Making the rules cheap

I built TestPerch — a free, MIT-licensed Chrome extension for Amplitude that turns the rules into clicks. It lives in Chrome's side panel and shows every manual Testing assignment in a project: for your account and for up to 20 custom emails, user IDs, or device IDs — exactly the alias workflow above. Join a test, switch variants, leave — per identity, without opening a single experiment page.

No API keys and no backend: it works through your already-open Amplitude session, so there's nothing for a security review to object to. There's a demo mode with fictional data — you can try the whole flow without an Amplitude account.

I'm the author; it's non-commercial and not affiliated with Amplitude. If your team has its own answer to variant amnesia — or TestPerch almost fits but misses something — I want to hear it. Bug reports and PRs welcome.

Top comments (0)