Running website A/B tests from Claude Code with an MCP server

#ai #mcp #claude #webdev

Coding agents can now ship a landing page in minutes. The slow part is finding out whether the new page actually converts better than the old one. That needs a split test on real traffic, which usually means a testing tool, a dashboard, and someone babysitting it.

Here is how to let the agent run that loop itself over MCP: add the site, install the tag, start a test, and read the winner, all from the terminal.

What MCP gives the agent

The Model Context Protocol lets a client (Claude Code, Claude, ChatGPT, Cursor, Codex) call tools on a remote server. For split testing, the useful tools are roughly:

setup: add a site, check the tag is installed
tests: create a test, list tests, read one test's numbers, change or stop it
evidence: heatmaps, session recordings, what past tests taught, what to test next
conversions: record a sale or sign-up from your backend

I will use Luria's MCP server because I build it, but the workflow is the same idea with any testing tool that exposes an API (PostHog has an MCP server for feature-flag experiments, for example).

1. Connect

claude mcp add --transport http luria https://www.luriart.com/api/mcp

The first call opens a browser: sign in with an emailed link and press Allow. No API key to paste. (Clients without OAuth can send Authorization: Bearer lk_... instead.) Cursor, Codex, VS Code and others take the same URL as a remote Streamable HTTP server:

{ "mcpServers": { "luria": { "url": "https://www.luriart.com/api/mcp" } } }

2. Add the site and the tag

Prompt:

Add example.com to Luria and install the tag in this repo's layout.

The agent calls add_site, gets back a one-line script tag and per-framework install notes, edits your layout file, then calls verify_install until the tag reports in. Until a site is connected, every other tool answers setup_needed with the exact next steps, so the agent cannot wander off and create tests on nothing.

3. Start a test

Start a test on the homepage headline. Keep the current one as the control.

The agent calls create_test (or auto_test to let Luria write the version from the page and the site's data). Two things worth knowing:

The first live change waits for the owner's OK. An agent can propose; a human approves the first one. After that, tests can run on their own.
The original stays live as a held-out control and never drops below a floor of traffic, so the result is measured against the same visitors at the same time, not a before/after guess.

4. Count real conversions

Clicks are not wins. If checkout happens on your backend (Stripe, a booking tool), send the conversion server-side:

curl -X POST https://www.luriart.com/api/v1/conversions \
  -H "Authorization: Bearer $LURIA_KEY" -H "Content-Type: application/json" \
  -d '{"site":"SITE_ID","visitor":"<luria_sid from the browser>","name":"purchase","value":49.99,"order_id":"1042"}'

Or just ask the agent: "Wire purchase conversions from our Stripe webhook to Luria." It has the record_conversion tool and your codebase.

5. Read the result

How is the headline test doing? If there is a winner, keep it and start the next test you suggest.

The agent calls get_test (visits, conversions and the chance each version beats the original), and next_tests / start_next_test to keep going. A change is kept only when it is 95% likely to beat the original after at least 200 visits per version. If you want to sanity-check how long that should take, here is a sample size calculator.

Guardrails I would insist on (with any tool)

A human approves the first live change.
A kill switch that stops every change fast.
Spend caps: prepaid credits or a hard monthly limit.
Read-only tools marked as read-only (readOnlyHint), so the client can auto-approve reads and ask for writes.
Errors that say what to send next ({"error", "code", "hint"}), so the agent fixes its call instead of guessing.

DEV Community