DEV Community

chovy
chovy

Posted on Originally published at dev.profullstack.com

A/B testing on w3bs.org, and OpenABTest 0.2

#api #opensource #softwaredevelopment #testing

W3BS now hosts A/B tests at w3bs.org/ab-testing. It is free, and it is built on OpenABTest, the open experiment format we publish on LogicSRC.

How it works

You define an experiment on the page: an id, two or more variants with weights and JSON parameters, and a goal such as signup-completed. Your server makes three calls with a key from the page:

curl -s https://w3bs.org/api/ab/assign \
  -H "Authorization: Bearer $W3BS_AB_KEY" -H 'Content-Type: application/json' \
  -d '{"experiment":"checkout-button","participant":"user-1842"}'
Enter fullscreen mode Exit fullscreen mode

assign returns the variant and its parameters. Then exposure when you showed it, and conversion when the goal happened. The same calls work from the terminal with w3bs ab in CLI 0.1.8.

What it refuses to do

Most A/B tools are generous with good news. This one is strict on purpose.

  • The variant is picked on the server and stored the first time. A reload, a second device or a cleared cookie never moves someone to the other arm.
  • Assigned, exposed and converted are separate counts. A conversion with no recorded exposure is shown, not dropped.
  • Zero participants reads "no data", not 0%.
  • Results show a 95% interval and a p-value against the control, and never a winner. A person decides.
  • A started experiment cannot be edited. To test something else, make a new one.

OpenABTest 0.2: goal conversions

OpenABTest 0.1 treated every conversion as a purchase with a price, which fits pricing experiments and nothing else. Version 0.2 adds goal conversions: a manifest names its goal, and a conversion carries either a price or that goal, with an optional observed value. Every 0.1 document is still valid. The schemas, validators and SDK are on npm as @logicsrc/schemas and @logicsrc/validators 0.5.0 and @logicsrc/sdk 0.4.0.

Every experiment on w3bs.org exports as an OpenABTest manifest and event stream, checked against those validators, so the data is yours to take anywhere.

Start at w3bs.org/ab-testing. The guide is at w3bs.org/docs/ab-testing.md.

Top comments (0)