DEV Community: The ABTestly team

Almost every A/B testing vendor refuses to publish a price. Here are ours

The ABTestly team — Sat, 26 Sep 2026 12:06:07 +0000

$99, $249, $549 a month.

Those are on our pricing page, and they are the least useful part of publishing a price.

The useful part is what publishing forces you to do, and what refusing to publish lets a vendor get away with.

Go and check

Open the pricing page of the major A/B testing platforms. Optimizely, VWO, AB Tasty.

You will find tiers with names, feature grids, and a button that says Contact Sales. What you will not find is a number at any traffic volume.

Convert publishes. Their Growth plan is $399 a month at 100,000 tested visitors, and you can read it without talking to anybody.

That is close to the whole market, and it is one of the odder norms in B2B software. Nobody hides the price of a database.

What a hidden price actually costs you

You cannot compare without entering a funnel. Evaluating four vendors means four discovery calls before you know whether any of them are in your range. That is not a scheduling annoyance, it is a deliberate cost imposed on comparison.

The price is a function of you, not the product. When the number arrives at the end of a call, it has been shaped by your company size, your stack, and how urgent you sounded. Two teams buying the same thing pay differently, and neither knows.

Renewal has no anchor. If the original number was negotiated rather than published, there is nothing to measure the increase against.

It selects for a particular kind of buyer. A team that can run a three-month procurement process is a different customer from a team that wants to try something on Thursday. Hidden pricing is how a vendor quietly stops selling to the second kind.

What publishing forces

It forces you to pick a metering unit and defend it.

Ours is monthly tested users, and the definition has edges we have to state rather than resolve in our favour. Subdomains of one registrable domain share a visitor id and count as one. Two genuinely different domains count as two, because browsers isolate storage per domain and there is no cross-domain identifier to join them.

That second line costs us money on any customer running a .com and a .co.uk. It is still what happens, so it is what the docs say.

It also forces you to stop selling features you do not ship at that tier. Frequentist statistics are on every plan including the $99 one. Sequential and Bayesian start at Pro, $249. I cannot write "three statistics engines" next to the $99 price, because that would be false, and the price being public is what makes it checkable.

Our terms, plainly

Plan	Price	Monthly tested users
Starter	$99	50,000
Pro	$249	200,000
Business	$549	500,000
Enterprise	Custom	Custom

Annual billing saves 20%.

Every account starts with a 14-day trial on a monthly plan. A card is required, and nothing is charged until day 15.

There is no free tier. There used to be, and it closed to new signups in September 2026. I would rather write that sentence than let someone discover it at signup.

The part that is not virtue

We can publish because we are small and we are not defending a renewal number against a board.

A vendor with a large enterprise base has a real reason to keep prices off the page: their existing contracts are all different, and publishing a list price makes every one of those negotiations legible at once. That is a genuine constraint, not just cowardice.

So this is a structural advantage rather than a moral one, and it will get harder to keep as we grow. Worth saying now, while it is true.

What to do with this

If you are evaluating tools, ask every vendor for a number at your actual traffic volume, in writing, before the demo.

The ones who give it to you have told you something real about how they will behave at renewal. The ones who will not have also told you something.

And check what the metering unit does with your specific setup, because that is where a published price quietly becomes a different price.

ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. The metering rules are documented at docs.abtestly.com, and the prices are at abtestly.com/pricing.

A rollback plan needs a detection plan

The ABTestly team — Sat, 26 Sep 2026 11:59:19 +0000

Every team running experiments has a rollback plan. Pause the test, ship the fix, move on.

Far fewer have a detection plan, which is the part that decides how long the broken thing was live before anyone hit the button.

A rollback you trigger on day nine is not a rollback. It is a post-mortem.

The release cadence changed under everyone

Chrome now ships a new Stable version every two weeks.

That is a two-week window between "this variation works" and "this variation runs on a browser that did not exist when it was written". Multiply by the other engines and their own cadences.

Most A/B test variations are DOM manipulation against selectors that were correct on the day they were authored. Nothing in that description survives contact with a fast release train automatically.

So the QA question stopped being "did it work when we built it" and became "how would we know if it stopped working".

What actually breaks, and when

The first version of a variation can look completely fine and still fail before the statistics ever run.

The failure modes are boringly consistent. A selector matches on first load and not after a client-side route change. A variation applies on desktop and silently no-ops on the mobile template. A goal fires on an element the variation replaced, so the exposure is recorded and the conversion never is.

None of those throw an error. None of them show up as a red thing on a dashboard. The test just quietly collects data that means nothing.

That is the detection gap. The variation is not down, it is wrong, and wrong looks identical to working until you check.

Editing a running test is the sharp edge

Here is the part most tools stay quiet about: a running test can be edited.

You found a bug in your variant code, you fix it, and now half your sample saw one thing and half saw another, reported as one number.

There are only two honest ways to handle that, and both of ours are confirmation-gated:

Pause, edit, resume. The clock resets on the results page and you get a new epoch. The old data is still there, it is just separated from what comes after.

Reset data. Discards the run so far and starts fresh with the updated code.

Reset is the heavier one, and it does more than clear counts. It rotates the experiment's bucketing salt, so visitors are assigned from scratch and someone who saw one variation may now see another.

That is the honest cost of a reset, and it is why it is not a shrug.

Two design decisions that follow from this

Material settings lock at 1,000 visitors.

Once a running experiment has 1,000 visitors since its last reset, six material settings become read-only in the editor:

Locked after 1,000 visitors. Pause the experiment to edit.

Pausing opens them again. The lock is not there to stop you, it is there to make the change deliberate, because changing traffic allocation mid-flight on a live test is how you manufacture a sample ratio mismatch.

The change notice does not go away.

If material settings changed during a run, the results page says so, and it keeps saying so until you reset the data.

That is deliberate. A notice that disappears once you have seen it is a notice that stops doing its job at the exact moment it matters, which is when a colleague reads the result three weeks later with no memory of what happened on day two.

Clearing it requires a reset, which advances the baseline so earlier changes fall outside the window. Nothing is deleted. The change events stay in the table, they just stop being in scope.

You cannot keep the numbers and lose the notice about them. That is the whole point.

Building an actual detection plan

Preview every variant on the live site before launching. Not in a staging clone, on the real page, because the thing that breaks is usually the interaction with production markup you did not write.

Watch the exposure count per arm in the first hour, not the conversion rate. Conversions are too sparse to tell you anything early. Exposures tell you immediately whether the variation is reaching people at the rate you expected.

Run the sample ratio check continuously, not at the end. An SRM that appears on day one is a bug you can still fix cheaply.

Decide in advance what number triggers a rollback, and write it down next to the hypothesis. "We will pause if exposures in variant B fall below 40% of control" is a detection plan. "We will keep an eye on it" is not.

And assume something will change underneath you in the next fortnight, because on the current release cadence, something will.

ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. The reset and locking behaviour is documented at docs.abtestly.com, and the prices are published at abtestly.com/pricing.

Your A/B testing tool and your analytics will never agree, and that is fine

The ABTestly team — Sat, 26 Sep 2026 11:57:42 +0000

"GA4 doesn't match, so I don't trust the test."

I hear this constantly, and the instinct behind it is right. Two systems reporting the same experiment should not disagree.

But they are not reporting the same thing, and once you see why, the disagreement becomes useful rather than alarming.

They are counting different populations

An A/B testing tool counts people it bucketed into an experiment. Analytics counts people who loaded a page and fired a tag.

Those two sets overlap. They are not equal, and nothing you configure will make them equal.

A visitor with an ad blocker that eats your analytics but not your testing snippet is in one and not the other. A visitor who leaves before your tag manager finishes is in neither, or in one. A bot filter that treats the two scripts differently splits them further.

So the question is never "why do the totals differ". It is "does the difference move with the variant", which is a completely different and much more answerable question.

The identity problem underneath it

Here is the mechanism that decides what "the same visitor" even means, and almost nobody checks it before reading a result.

The visitor id lives in one domain's localStorage and cookie jar. That single fact produces this:

Where the same person browses	Counted as
`example.com` and `shop.example.com`	1 visitor
Two experiments on the same site	1 visitor
`site-A.com` and `site-B.com`	2 visitors
The same domain in two different organisations	2 visitors

Subdomains of one registrable domain share a visitor id. Two genuinely different domains cannot, because browsers give each domain its own isolated storage.

The same human arrives as two unrelated visitors, and is metered twice.

There is no cross-domain identifier, and we are not planning to add one. That is a deliberate position, not a gap in the roadmap. Stitching identity across domains means shipping something that follows people between sites, and we would rather explain the limitation than build that.

If your brand runs example.com and example.co.uk as separate registrable domains, your testing tool sees two audiences. So does your analytics, differently. That is most of the gap right there.

How to actually reconcile them

Stop comparing totals. Join on the event.

GA4 emits an experience_impression event carrying exp_variant_string. Pipe GA4 to BigQuery and join that event to your conversion events on user_pseudo_id.

That is the same join GA4 does internally, except in raw SQL, with no Custom Dimension cap, no Audience limits, and no (other) rollup swallowing your variant names.

The (other) bucket is the one that catches people out. GA4 has cardinality limits, and when you exceed them it quietly rolls values up. Your variant B stops being variant B and becomes part of a bucket, and the report still renders, and nobody notices.

For concurrent experiments this stops being optional. Custom Dimensions run out fast, and the in-product GA4 reporting cannot express "bucketed into experiment 1 variant B and experiment 2 control".

What we keep, and for how long

The ledger is the canonical store for an experiment. Every exposure and every goal that was accepted, with its payload.

Behind it sits a raw compressed event archive, and that archive is purged nightly once it passes 24 months. It is a data-retention policy, not a billing lever, and it is unaffected by your plan or your subscription state.

The ledger the results are computed from survives the purge and stays on the results page for the life of the experiment. Only the raw events age out.

If you need raw events beyond 24 months, run the async export inside the window and keep the CSV in your own storage. We would rather tell you that now than when you go looking for month 25.

The practical rule

Pick one system as the system of record for the decision, before the test starts. For an experiment, that should be the testing tool, because it is the only one that knows who was bucketed.

Use analytics for the diagnosis, not the verdict. It is excellent at telling you where a difference came from and poor at telling you whether the difference is real.

And when the numbers disagree, check the direction of the gap rather than its size. A gap that is the same shape in both arms is measurement. A gap that differs by variant is a bug, and it is the most important thing on your screen.

ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. The GA4 and BigQuery join is documented at docs.abtestly.com, and the prices are published at abtestly.com/pricing.

How long should an A/B test run? Not a number of weeks

The ABTestly team — Sat, 26 Sep 2026 11:50:44 +0000

Two quantities decide how long a test runs, and the calendar is not one of them.

How many visitors per variation the test needs, and how fast you supply them. That is the whole projection.

Our duration estimator returns 15 days for a particular test. The schedule you should commit to is 21.

The gap between those two numbers is the interesting part, and it is where most test plans go wrong.

What the projection actually computes

Required sample per arm, divided by arrival rate. That is it.

required is always computed at 95% confidence and 80% power. Not at whatever you set in the experiment. At 95 and 80, every time.

So if you configured the test at 99% confidence, or you are running four variations under a Bonferroni or Sidak correction, you need more than the projection says.

Treat the number as a floor. It is the arithmetic, not the plan.

Four things it does not know

It does not round to whole weeks.

Behaviour varies across the week. A run that stops mid-week hands one variation an extra Saturday.

Round your own end date up to a whole number of weeks so every day of the cycle appears the same number of times in both arms. That is why 15 becomes 21.

It does not hold a business-cycle floor.

If traffic is heavy enough that the sample arrives in two days, the projection will happily say two days.

Two days is one narrow slice of your audience and one mood of the market. Hold a floor of at least one full cycle, preferably two, so you can watch the effect survive a second week.

If your purchase cycle is longer than a week, stretch the floor to match it.

It does not model novelty or primacy.

Returning visitors react to a change because it is new. Some engage more than they will once it is ordinary. Some are thrown by an unfamiliar layout.

Both effects fade. Neither is visible to a projection that only counts visitors.

It does not know your confidence level or your correction.

Covered above, and it is the one people are most surprised by.

The trap at the other end

There is a status on the runway line called alreadySignificant, and it short-circuits the projection to reached.

The moment the leading variation clears 95% on the gated p-value, the line stops showing a date and starts saying you have enough.

On day 4 of a planned three-week run, that is exactly the reading a fixed-horizon test cannot support.

So we wrote the caveat into the product rather than leaving it in a blog post. The status describes the state of the evidence right now. It is not permission to stop.

If you want a runway you are allowed to watch continuously, that is what a sequential test is for. A fixed-horizon test plus a live dashboard is not the same thing, however much it looks like it.

Why we gate small numbers instead of hiding them

There is a floor of 100 visitors and 5 conversions before certain figures are judged at all.

But the rate, the interval and the lift are never withheld, at any sample size.

Those three are honest at any n, because the interval is the thing that says so. A 95% interval of [-19%, +44%] is not a weak result being hidden from you. It is the result.

What we refuse to do is print a bare percentage off a handful of visitors, because a bare percentage is the thing people screenshot.

The revenue-per-visitor interval uses the same bar, since a revenue interval is at its noisiest exactly where that bar is not met.

One thing a confidence interval is not

It is not the probability that the true value lies inside it.

Under the frequentist reading, the true value is fixed and the interval is what is random. A 95% interval is a procedure that brackets the truth in 95% of repetitions.

The sentence people actually want sounds like "there is an 87% probability this variation is better". That is a posterior probability with a credible interval beside it, and it comes from a Bayesian engine, not from a frequentist CI with the words swapped.

It is also not a guard against bias. An interval describes sampling noise and nothing else. A test with a sample ratio mismatch can have a beautifully tight interval around a meaningless number.

The practical version

Before you launch, compute the required sample at the confidence and power you are actually using, including any correction.

Divide by your real arrival rate, not your best week.

Round up to whole weeks. Apply a floor of one full business cycle, two if you can afford it.

Write the end date down before the test starts, and treat every look before it as monitoring rather than deciding.

Then, when someone asks on day 4 whether it is winning, the answer is that the test is still running, and that is a complete answer.

ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. The duration model and its caveats are documented at docs.abtestly.com, and the prices are published at abtestly.com/pricing.

2.2 ms, 19.9 ms, 529 ms: benchmarking an A/B testing runtime and publishing the caveats

The ABTestly team — Sat, 26 Sep 2026 11:24:22 +0000

Most A/B testing vendors quote one performance number, and it is the wrong one.

The number is snippet size. Ours is about 31 KB gzipped. That figure tells a visitor nothing at all about what they saw, or when.

So here is the measurement we actually take, the method behind it, and the reasons it is a floor rather than a result.

The numbers

p75 time from navigation start to a variation landing in the DOM, over 200 page loads in each condition:

Condition	p75
Single page app route change	2.2 ms
Repeat view, warm cache	19.9 ms
First visit, empty cache, throttled	529 ms

The throttle is 1.6 Mbps down with a 150 ms round trip. The test runs in headless Chromium against the real built runtime, with one variation that rewrites a heading above the fold.

Separately, because it is the thing a visitor actually experiences: on that throttled first visit the original heading was on screen first on all 200 loads, for a median of 326 ms.

On a cached repeat view, no painted frame showed the original on 199 of the 200 loads. On the route change, none did at all.

Why 529 ms is a floor and not a result

The measurement runs against a local origin.

That means it carries the throttle and the transfer, but none of the DNS, TLS or edge latency a real first visit also pays.

It also throttles the network without throttling the processor. A first visit on a mid range phone is slower than 529 ms, not faster.

We would rather say that than let someone quote 529 ms at a prospect as a field figure.

Why the page flickers at all

The runtime is injected as a dynamic <script>. It does not block the parser.

That is a deliberate choice, and the cost of it is the 326 ms on that first visit. The original renders, then the variation replaces it.

The usual fix is an anti-flicker overlay that hides the page while the runtime decides. Ours exists, and it is unchecked by default.

Checked, the snippet adds a rule and puts the class on <html> at the top of the page:

.abtestly-async-hide { opacity: 0 !important }

The class comes off once variants apply, or after two seconds, whichever is first. The page never stays hidden.

The reason it is off by default is the arithmetic of who pays for it.

A page-wide hide costs every visitor on every page, including everyone who was never bucketed into any experiment. If the config fetch is ever slow, that is a visibly empty site.

Flicker only shows on the pages a variant actually changes. So the box is there for the test that needs it, and not before.

There is a second reason, and it is the one that makes this more than a preference. Our own speed guardrail flags a variation as slower when its p75 largest contentful paint sits 400 ms or more above control. An always-on overlay would ship exactly the regression that guardrail exists to catch.

Where the variant decision happens

In the browser, after the runtime arrives. There is no server hop in the path the benchmark measures.

Assignment is a deterministic hash with no shared state, so every visitor's bucket is computed from their own id alone:

hash1 = murmur3(salt + "/" + userId + "/Exposure") mod 10000
enter = hash1 < trafficAllocationBp

salt is the experiment's immutable UUID, minted at creation and never changed. userId is a UUIDv7 held in localStorage and a cookie. trafficAllocationBp is allocation in basis points, so 10000 is 100%.

Entry is decided first, variant second. The same two-step model Amplitude uses.

Because it is a pure function of those inputs, the same hash on the same inputs reproduces the same bucket in a Node script. You can check our assignment against your own implementation without asking us anything.

The guardrail admits what it is not

This is the part I want to hold up, because it is written in our own docs about our own feature rather than left for a prospect to discover:

There is no confidence interval, no bootstrap, and no significance test. The panel is a descriptive guardrail, not an inferential one.

So it reports a measurement. It does not prove a variation is safe.

The docs volunteer the consequence too. A difference just over 400 ms on 100 page loads per arm is not the same evidence as the same difference on 100,000. Same flag, very different weight.

The panel does not weight them for you. The page load count sits next to the row so that you can.

Four checks run before a row is judged at all, and they stop at the first failure:

capture rate not above 100%
at least 100 page loads on both arms
a capture rate gap of no more than 20 percentage points
at least 50% capture on each arm

Fail one and the row reads "collecting speed data" or "comparison unavailable", depending on which check it was, rather than guessing.

There is also no correction across devices or variations. A four variant test on two devices produces six comparisons, each judged on its own threshold. Six independent calls, not one corrected verdict.

That is the discipline a descriptive guardrail asks of you. Read the page load count behind a flag before acting on it, and read "not flagged" as nothing visible at this volume, rather than as clearance.

What to ask your own vendor

Not snippet size.

Ask for p75 time from navigation start to the variation landing in the DOM, separated by cached and uncached. Ask how long the original was actually painted. Ask for the method, and for the caveats.

If a vendor cannot produce that, the honest reading is that nobody there has measured it.

ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. The bucketing algorithm is documented at docs.abtestly.com and the prices are published at abtestly.com/pricing.

Why our A/B testing tool refuses to call a winner

The ABTestly team — Sat, 26 Sep 2026 11:10:39 +0000

Two readings of the same experiment.

Day 4. Variant B is up 12.7% on conversion rate. 12,506 visitors in the variant, 12,418 in control. The number is green. Somebody screenshots it and drops it in Slack.

Day 11. Variant B is up 12.7%. Same test, same split, same traffic mix.

Same experiment, same lift. The only thing that changed is how long it was allowed to run. One of those two readings is a result. The other is a coin that has come up heads four times.

Every dashboard I have used will render both of them identically, with a green arrow.

The thing that goes wrong is peeking

A standard frequentist A/B test is a fixed-horizon test. You commit to a sample size before you start, you look once when you reach it, and the α you chose (usually 0.05) is the probability of a false positive for that one look.

The moment you look more than once, that guarantee is gone.

This is not a subtle effect and it is not our finding; it is the classic result on repeated significance testing. Each additional look is another chance for the random walk of the difference between two proportions to wander across your threshold. Look ten times at α = 0.05 and the real false-positive rate lands far closer to 20% than to 5%. Monitor continuously and, in the limit, you will cross the line eventually with probability approaching 1, whether or not there is any effect at all.

Now think about how experimentation actually works in a company. The test is live. The dashboard is open in a tab. There is a standup every morning and a roadmap review on Thursday. Nobody looks once.

So the tooling is not neutral here. A dashboard that renders a day-4 lift the same way it renders a day-11 lift is not reporting a result, it is handing someone a screenshot to win an argument with.

What we show instead

Three decisions, all of which make our product look worse in a demo.

1. The verdict panel says "Still collecting" and means it.

Not "leading". Not "trending positive". Until the decision boundary is crossed, the headline on the results page is that the test has not concluded, and the lift figure sits underneath it rather than above it. Whoever opens that page has to read the word before they read the number.

2. The confidence interval is always shown next to the sample size.

A lift is meaningless on its own. +12.7% with a 95% interval of [-2.1%, +28.4%] at n = 12,506 is a different object from +12.7% with [+8.9%, +16.6%] at n = 180,000, and the only way to stop people conflating the two is to refuse to print one without the other. No bare percentages anywhere in the UI.

3. Sample ratio mismatch is on the results page, not in a settings drawer.

If you asked for 50/50 and you are getting 50.4/49.6 across 200,000 visitors, something upstream is broken: a redirect that fires before the assignment, a bot filter that treats variants differently, a cache that serves control to a subset. The check is a chi-square goodness of fit against the expected allocation, and a low p-value there invalidates everything above it on the page.

SRM is the most useful check in experimentation and the one most often buried. If the split is broken, the lift is not wrong, it is meaningless. We put the flag at the top of the result, in red, before the number.

Sequential is the real answer to peeking

Telling people "don't look" loses to reality every time. The better answer is to use a method designed for looking.

Sequential tests spend your error budget across the monitoring period rather than at a single endpoint, so continuous monitoring is valid by construction rather than in spite of the maths. You give up some power relative to a perfectly executed fixed-horizon test, and in exchange you get a test that survives the way teams actually behave.

Our default engine is frequentist because that is what most teams already reason in. Sequential and Bayesian are available on the higher plans for teams that want to monitor continuously or reason about probability-to-beat-control directly. Whichever engine you pick, the method version is locked when the test starts, so nobody can switch engines halfway and pick the flattering one.

Why any vendor would build this

Here is the part that is uncomfortable to write.

Every incumbent's renewal conversation depends, in some form, on the customer believing their testing programme is working. A dashboard that surfaces wins is commercially aligned with that. A dashboard that says "still collecting" for nine days is not. "Still collecting" is a worse demo than a trophy, and I have watched it be a worse demo.

We can afford to build it because we are small, we publish our prices, and we are not defending a renewal number against a board. That is a structural advantage rather than a virtuous one, and it will not last forever.

But if you sell a statistics product, refusing to overclaim is the product. Everything else is a rendering detail.

ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. Docs at docs.abtestly.com, and the pricing is published at abtestly.com/pricing.