DEV Community

Cover image for Chrome Extension A/B Testing in MV3 Without Breaking Policy
ExtensionBooster
ExtensionBooster

Posted on Originally published at extenify.app

Chrome Extension A/B Testing in MV3 Without Breaking Policy

Originally published on Extenify.

Chrome extension A/B testing sounds simple until you hit the first wall: the Chrome Web Store reviews every code change, and Manifest V3 forbids loading code from your server. The good news is that a remote config that only switches between variants you already shipped is explicitly allowed by policy, and with a hashed assignment per install plus one tracked metric you can run real experiments inside a reviewed extension.

The rule that makes it work: the config chooses, the package decides. Every variant's code ships in your build; the remote config only turns experiments on or off and changes their weights. The Chrome Web Store's Manifest V3 requirements state that "Fetching a remote configuration file for A/B testing" is allowed "where all logic for the functionality is contained within the extension package." What gets you rejected is anything that smells like remote execution: remote <script> tags, eval() of fetched strings, or a config that describes behavior in enough detail to act as an interpreter. Firefox says the same thing ("Add-ons must be self-contained and not load remote code for execution"), so one policy-safe design passes review on Chrome, Edge, and AMO.

The setup in four steps

  1. Give every install a stable ID. On chrome.runtime.onInstalled, write crypto.randomUUID() and Date.now() into chrome.storage.local. The ID survives restarts and updates; the timestamp lets you enroll only installs that happened after an experiment started.
  2. Host a tiny config file. It only toggles and weights:
{
  "onboarding_checklist": { "active": true, "weights": [50, 50], "startedAt": "2026-10-20T00:00:00Z" },
  "popup_upgrade_copy": { "active": false }
}
Enter fullscreen mode Exit fullscreen mode
  1. Hash for assignment. SHA-256 the experiment name plus the install ID, take the first 4 bytes modulo 100, and map the bucket to a variant using the config weights. Store the assignment: it wins over later weight changes, which keeps your results clean.
  2. Fail toward control. No ID, no network, or "active": false all return the control variant. Cache the config for a few hours so the service worker does not refetch constantly, and put your config URL in host_permissions.

Log exposure, not assignment

Assignment is not the same as seeing. If 70% of new users never open the onboarding page, analyzing everyone who was assigned dilutes your result with people who saw neither variant. Fire an experiment_exposure event the first time a user actually renders a variant, and analyze exposed users only.

Pick one primary metric before you launch and one guardrail. For extensions the useful primary metrics are activation (the key action within 24 hours), day-7 retention, and upgrade rate. The guardrail that matters most is uninstall rate: pass your assignments through chrome.runtime.setUninstallURL() so uninstalls are counted per variant, and watch new reviews during the test — an annoying variant shows up in the rating long before it shows up in a significance test.

Size the test before you start

Small extensions run out of users before they run out of ideas. Detecting a 40% to 45% activation lift at 95% confidence and 80% power needs 1,534 users per variant; a 25% to 28% retention change needs 3,397; moving upgrades from 2% to 2.5% needs 13,809. At 200 new installs a day split 50/50, the first test takes about 16 days of enrollment, plus another 7 if your metric is day-7 retention. If the math says months, test a bigger change: a new onboarding flow can move activation by ten points; a button color will not.

Baseline Target Users per variant
40% activation 45% 1,534
40% activation 44% 2,389
25% day-7 retention 30% 1,251
2% upgrade rate 3% 3,826

When the test ends, read the confidence interval, not just the p-value. A 6.0-point lift with a 95% confidence interval of 2.5 to 9.5 points and p = 0.0008 is a clear win; a 3.3-point lift with an interval of −0.2 to 6.8 points (p = 0.06) is "probably positive, not proven." Do not stop early because the numbers look good — that inflates false positives — and keep new releases out of the test window. When a variant wins, set its weight to 100 so new installs get it immediately, make it the default in the next release, and delete the losing code path.

Key takeaways

  • Remote config for A/B testing is explicitly allowed in Manifest V3, as long as all logic ships in your package and the config only switches or weights variants.
  • Assign with a hash of experiment name plus install ID, store the result, and enroll only installs that came after the experiment started.
  • Track exposure events and one primary metric, with uninstall rate as a guardrail.
  • Sample sizes: 1,534 users per variant for 40% to 45% activation, 3,397 for a 5-point retention change, 13,809 for a 2% to 2.5% upgrade lift.
  • The Chrome Web Store listing itself cannot be split-tested; unlike Google Play and the App Store, there is no listing experiment feature.

For the complete experiments.js module, the two-proportion significance test, and the full sample-size table, see the full guide: Chrome extension A/B testing in MV3. If you ship extension or app experiments, Extenify records users, ratings, reviews, and versions every day, so you can check whether an experiment's win still holds a month later.

What is the smallest experiment you have run in an extension, and did the confidence interval agree with what the reviews said?

Top comments (0)