DEV Community: Baneet Singh

Your Shopify price test is picking the wrong winner

Baneet Singh — Sat, 31 Oct 2026 02:06:30 +0000

Most price tests on Shopify pick the wrong winner, and the reason is the metric, not the tooling.

A lower price almost always converts better. If you judge a price test on conversion rate, the cheapest price wins every time, even when it makes you less money.

Judge price tests on profit per visitor

profit per visitor = conversion rate × (price − COGS − shipping − fees)

A worked example with $15 of cost per order:

Price	Conversion rate	Profit per order	Profit per visitor
$40	3.0%	$25	$0.75
$48	2.7%	$33	$0.89

The $48 price converts 10% worse and earns 19% more per visitor. A conversion-rate dashboard would have told you to keep $40.

Here is the same math as a tiny function you can drop into a notebook:

const profitPerVisitor = (cr, price, cost) => cr * (price - cost);
profitPerVisitor(0.03, 40, 15);  // 0.75
profitPerVisitor(0.027, 48, 15); // 0.891

Three ways to run the test on Shopify

Duplicate variants. Each price is a hidden copy of the variant and an app shows one copy per visitor. Easy, but inventory and reports split across the copies.
Shopify Functions (Plus only). A cart transform sets the price at checkout. Clean data, needs Plus and a developer or an app.
Separate offer pages. Two landing pages with different prices or bundles, traffic split by link. Only works for paid traffic you can route.

Rules that keep it honest

One visitor sees one price for the whole test, on every page and at checkout.
Charge the price that was shown. Never raise it at checkout.
Run whole weeks. Price tests need more traffic than page tests because the profit gap is small.
Check local consumer law before showing different prices to different people.

Sample size

Profit differences are small, so you need more orders than you think. At a 2% conversion rate, detecting a 20% difference takes roughly 21,000 visitors per version at 95% confidence and 80% power. There is a free calculator and table here.

I build Luria, which runs split tests on the parts of a page around the price (photos, offer wording, buy button, reviews placement), judged on real orders. For changing the price itself, a dedicated price testing app is the right tool. Full guide: How to A/B test prices on Shopify.

Running website A/B tests from Claude Code with an MCP server

Baneet Singh — Sat, 31 Oct 2026 01:30:18 +0000

Coding agents can now ship a landing page in minutes. The slow part is finding out whether the new page actually converts better than the old one. That needs a split test on real traffic, which usually means a testing tool, a dashboard, and someone babysitting it.

Here is how to let the agent run that loop itself over MCP: add the site, install the tag, start a test, and read the winner, all from the terminal.

What MCP gives the agent

The Model Context Protocol lets a client (Claude Code, Claude, ChatGPT, Cursor, Codex) call tools on a remote server. For split testing, the useful tools are roughly:

setup: add a site, check the tag is installed
tests: create a test, list tests, read one test's numbers, change or stop it
evidence: heatmaps, session recordings, what past tests taught, what to test next
conversions: record a sale or sign-up from your backend

I will use Luria's MCP server because I build it, but the workflow is the same idea with any testing tool that exposes an API (PostHog has an MCP server for feature-flag experiments, for example).

1. Connect

claude mcp add --transport http luria https://www.luriart.com/api/mcp

The first call opens a browser: sign in with an emailed link and press Allow. No API key to paste. (Clients without OAuth can send Authorization: Bearer lk_... instead.) Cursor, Codex, VS Code and others take the same URL as a remote Streamable HTTP server:

{ "mcpServers": { "luria": { "url": "https://www.luriart.com/api/mcp" } } }

2. Add the site and the tag

Prompt:

Add example.com to Luria and install the tag in this repo's layout.

The agent calls add_site, gets back a one-line script tag and per-framework install notes, edits your layout file, then calls verify_install until the tag reports in. Until a site is connected, every other tool answers setup_needed with the exact next steps, so the agent cannot wander off and create tests on nothing.

3. Start a test

Start a test on the homepage headline. Keep the current one as the control.

The agent calls create_test (or auto_test to let Luria write the version from the page and the site's data). Two things worth knowing:

The first live change waits for the owner's OK. An agent can propose; a human approves the first one. After that, tests can run on their own.
The original stays live as a held-out control and never drops below a floor of traffic, so the result is measured against the same visitors at the same time, not a before/after guess.

4. Count real conversions

Clicks are not wins. If checkout happens on your backend (Stripe, a booking tool), send the conversion server-side:

curl -X POST https://www.luriart.com/api/v1/conversions \
  -H "Authorization: Bearer $LURIA_KEY" -H "Content-Type: application/json" \
  -d '{"site":"SITE_ID","visitor":"<luria_sid from the browser>","name":"purchase","value":49.99,"order_id":"1042"}'

Or just ask the agent: "Wire purchase conversions from our Stripe webhook to Luria." It has the record_conversion tool and your codebase.

5. Read the result

How is the headline test doing? If there is a winner, keep it and start the next test you suggest.

The agent calls get_test (visits, conversions and the chance each version beats the original), and next_tests / start_next_test to keep going. A change is kept only when it is 95% likely to beat the original after at least 200 visits per version. If you want to sanity-check how long that should take, here is a sample size calculator.

Guardrails I would insist on (with any tool)

A human approves the first live change.
A kill switch that stops every change fast.
Spend caps: prepaid credits or a hard monthly limit.
Read-only tools marked as read-only (readOnlyHint), so the client can auto-approve reads and ask for writes.
Errors that say what to send next ({"error", "code", "hint"}), so the agent fixes its call instead of guessing.

Thompson sampling for A/B tests in 40 lines of JavaScript (and the trap nobody mentions)

Baneet Singh — Sat, 31 Oct 2026 01:25:45 +0000

A classic A/B test splits traffic 50/50 until it reaches a fixed sample size. A multi-armed bandit moves traffic toward the version that is winning while the test runs, so fewer visitors see the loser. Thompson sampling is the simplest good way to do it.

Here is a working implementation with zero dependencies, a simulation, and the one trap that makes a lot of bandit "wins" impossible to prove.

The idea in one paragraph

For each version, keep a belief about its conversion rate: a Beta distribution built from its visits and conversions, Beta(1 + conversions, 1 + visits - conversions). For every new visitor, draw one plausible rate from each version's belief and show the version with the highest draw. Versions you are unsure about still get explored; versions that are clearly worse get picked less and less.

The code

JavaScript has no built-in Beta sampler, so we build one from two Gamma draws (Marsaglia-Tsang):

function normal() {
  const u = 1 - Math.random(), v = Math.random();
  return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
}

// Gamma(k, 1), Marsaglia-Tsang; boost trick for k < 1.
function gamma(k) {
  if (k < 1) return gamma(k + 1) * Math.random() ** (1 / k);
  const d = k - 1 / 3, c = 1 / Math.sqrt(9 * d);
  for (;;) {
    let x, v;
    do { x = normal(); v = 1 + c * x; } while (v <= 0);
    v = v * v * v;
    const u = Math.random();
    if (u < 1 - 0.0331 * x ** 4 || Math.log(u) < 0.5 * x * x + d * (1 - v + Math.log(v))) return d * v;
  }
}

const beta = (a, b) => { const x = gamma(a); return x / (x + gamma(b)); };

// Pick a version for the next visitor.
function pick(arms) {
  let best = 0, bestDraw = -1;
  arms.forEach((arm, i) => {
    const draw = beta(1 + arm.conversions, 1 + arm.visits - arm.conversions);
    if (draw > bestDraw) { bestDraw = draw; best = i; }
  });
  return best;
}

// Chance that B beats A, by simulation.
function chanceBeats(a, b, draws = 20000) {
  let wins = 0;
  for (let i = 0; i < draws; i++)
    if (beta(1 + b.conversions, 1 + b.visits - b.conversions) >
        beta(1 + a.conversions, 1 + a.visits - a.conversions)) wins++;
  return wins / draws;
}

Simulate it

Two versions, true conversion rates 2.0% and 2.5%, 40,000 visitors:

const truth = [0.02, 0.025];
const arms = truth.map(() => ({ visits: 0, conversions: 0 }));
for (let t = 0; t < 40000; t++) {
  const i = pick(arms);
  arms[i].visits++;
  if (Math.random() < truth[i]) arms[i].conversions++;
}
console.log(arms, chanceBeats(arms[0], arms[1]));

Two runs on my machine:

A: 7,745 visits, 160 conversions   B: 32,255 visits, 801 conversions   P(B>A) = 0.984
A: 2,278 visits,  41 conversions   B: 37,722 visits, 975 conversions   P(B>A) = 0.992

The bandit found the better version and sent it most of the traffic. A 50/50 test would have shown the worse page to 20,000 people.

The trap: you starve your control

Look at the second run: the original page got only 2,278 visits. That is fine when you only care about earning more during the test, but it is a problem when someone asks "did the change actually help, and by how much?" With so few visits on the original, the estimate of the original's rate is wide, and seasonality or an ad change can fool you.

The fix is boring and important: keep a floor on the original. Never let it drop below some share of traffic (10% to 20% is common), and only call a winner after a minimum number of visits per version (a few hundred at least) and a high bar (for example 95% chance to beat the original).

function pickWithFloor(arms, floor = 0.1) {
  if (Math.random() < floor) return 0; // arm 0 = your unchanged page
  return pick(arms);
}

When to use which

Fixed 50/50 test: you need a precise size of effect for a big decision (pricing, a redesign) and will reuse the learning for months.
Bandit with a floor on the original: everyday page copy, where earning more during the test matters and you still want proof against the unchanged page.

If you want the sample-size side of this, I wrote that up too: How many visitors does an A/B test actually need? and a longer comparison of bandits vs A/B tests.

Disclosure: I build Luria, which runs Thompson sampling with a floor on the original page for store split tests. The code above is standalone and MIT-style; use it anywhere.

How many visitors does an A/B test actually need? (table + free calculator)

Baneet Singh — Sat, 31 Oct 2026 01:24:47 +0000

Most A/B tests that "won" never had enough visitors to win anything. Here is the math, a table you can bookmark, and a tiny zero-dependency calculator.

The short answer

At a 2% conversion rate, detecting a 20% relative lift (2.0% to 2.4%) takes about 21,106 visitors per version at 95% confidence and 80% power. Two versions means about 42,000 visitors on that page.

Smaller lifts and lower conversion rates need a lot more:

Current conversion rate	Detect +10%	Detect +20%	Detect +30%	Detect +50%
1%	163,092	42,691	19,824	7,747
2%	80,679	21,106	9,795	3,823
3%	53,208	13,911	6,452	2,515
5%	31,231	8,155	3,778	1,468
10%	14,749	3,839	1,772	707

Visitors per version. Two-sided test, 95% confidence, 80% power.

The formula

For two proportions p1 (current) and p2 (with the lift):

n = (z_a/2 + z_b)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2

with z_a/2 = 1.96 for 95% confidence (two-sided) and z_b = 0.84 for 80% power.

In JavaScript:

const Z = 1.959964 + 0.841621;
function sampleSize(base, lift) {
  const p2 = base * (1 + lift);
  return Math.ceil((Z * Z * (base * (1 - base) + p2 * (1 - p2))) / (p2 - base) ** 2);
}
sampleSize(0.02, 0.2); // 21106

I put this (plus an inverse-normal function so you can change confidence and power, and a CLI) in a small MIT repo: baneets/ab-test-sample-size.

npx github:baneets/ab-test-sample-size 2 20 50000
# 21,106 visitors per version (42,212 total), 95% confidence, 80% power
# about 3.7 weeks at 50,000 visitors a month

How long that takes

A page with 50,000 visitors a month at 2% needs about 3.7 weeks to detect a 20% lift, and about 14 weeks for a 10% lift. At 10,000 visitors a month, the 20% case takes about 18 weeks.

That is why button-colour tests on low-traffic stores almost never produce a real answer.

Three mistakes that create fake winners

Peeking. Checking a fixed-sample test every day and stopping the first time it looks significant inflates false positives a lot. Either commit to the sample size, or use a method built to be checked as it runs (sequential tests, Bayesian tests, multi-armed bandits).
Tiny effects on small traffic. If your page gets 5,000 visits a month, test big changes (offer, headline, first screen), not small ones. Big lifts need far fewer visitors.
Not running whole weeks. People buy differently on Monday and Saturday. Run at least one full week even if the number is reached sooner.

Bandits vs fixed tests

A classic A/B test splits traffic 50/50 until the sample is reached. A multi-armed bandit (often Thompson sampling) moves traffic toward the version that is winning while the test runs, so fewer visitors see the loser. The catch: a pure bandit can starve your original page, which makes the gain hard to prove. The fix is to keep a holdout share on the original no matter what.

I wrote more on that here: multi-armed bandit vs A/B testing.

Try it in the browser

If you just want the number without installing anything, there is an interactive version here: A/B test sample size calculator.

Disclosure: I build Luria, an AI split-testing tool, which is why I care about this. The math above is standard and works with any testing tool.