DEV Community

loading...

Discussion on: A/B tests for developers

Collapse
timkor profile image
Timkor • Edited

We use cookies to set the variant within the load balancer. The instances run within the same server so the only latency difference should be a result of changes to the code. Which is exactly what you want to test.

Also, how are cookies different than your approach?

Thread Thread
bgadrian profile image
Adrian B.G. Author

I said users lifetime (across sessions). Cookies are session based/volatile, if the user uses other browser or cleans its cookies they will see different version, which result in the previous issues I mentioned.

I did not presented my approach because I don't know all the details, but all the tests we did in gaming were using a database for persistence, we had the luxury of having all our visitors authenticated users so we know who is in what test.

Depending on what is tested persistence is required or not. Usually our features require a few weeks of measuring its impact on the user behavior, without authentication this cannot be done properly.

Thread Thread
timkor profile image
Timkor

Nevertheless, this would also be possible using load balancing, although it might require some customization.

Anyway, I use it for webshops, targeting new users. Most e-commerce websites do not have the luxery to use authentication before any actual conversion. Then this approach will work perfectly.

Good article though.

Thread Thread
bgadrian profile image
Adrian B.G. Author

When testing an action like Conversion sounds great, you do not even care about user as a lifetime, but rather to which version reacted better/got converted.

Beside Round Robin, the LB/Proxy can also be used to make the cohorts, for example based on country, or limit an entire Test based on a property (example country, region, language, device, mobile vs web).

Forem Open with the Forem app