A/B tests your coding agent can run

Andrei Babrounik — Wed, 14 Oct 2026 18:14:32 +0000

Your coding agent can change a headline in seconds. Finding out whether that headline helps people sign up takes an experiment.

I built SplitHub to keep that experiment in the same workflow as the code. You declare variants and a goal in splithub.config.ts, render the assigned value in your app, and read results from the CLI or an MCP client.

Start in your app

SplitHub needs Node.js 22.13 or newer. Run:

npx splithub init

For your own database, use:

npx splithub init --mode self

The React SDK provides SplitHubProvider, useVariant, useValue and useTrack. The Next.js package adds middleware and server component helpers. Assignment uses a local hash, so choosing a variant doesn't require a network request during rendering.

Give your agent the tools

To connect the MCP server in Claude Code:

claude mcp add splithub -- npx -y @splithub/mcp

For another MCP client, the configuration is:

{
  "mcpServers": {
    "splithub": {
      "command": "npx",
      "args": ["-y", "@splithub/mcp"]
    }
  }
}

You can then ask it to create a headline test, start or pause an experiment, push config, or read results. The server uses the same project config and credentials as the CLI.

For example:

Test a shorter headline on the pricing page, using signup as the goal.

Later:

Read the results for the pricing headline test. Is there enough evidence to choose a variant?

The CLI also lets you inspect results directly:

npx splithub status
npx splithub results hero-title

Replace hero-title with your experiment key.

Cloud or your own storage

SplitHub Cloud returns statistical verdicts. Self-hosted storage returns raw visitors, conversions, conversion rates and revenue. The public SDK is MIT licensed; the cloud analysis is separate.

You still need real visitors and a measured goal. An agent can help operate a test, but it can't replace evidence with a confident answer.

The repository includes complete Vite + React and Next.js examples.

Explore the SDK on GitHub · Documentation

Questions or feedback: info@splithub.io

DEV Community: Andrei Babrounik

A/B tests your coding agent can run

Start in your app

Give your agent the tools

Cloud or your own storage