DEV Community

Cover image for Unit Testing A/B Test Code

Posted on

Unit Testing A/B Test Code

Now, this seems like something that should be pretty straight forward ... testing the test and all that; what-what. But there are a few underlying considerations that make how to Unit Testing A/B Test Code a fairly remarkable question.

Common Language

While writing this article about testing test code, the language started to get a bit wonky.

Here are the definitions I will use moving forward ...

  • A-TEST CODE refers to the code that should already exist when it is decided that and A/B Test is needed. The expectation should be that Unit and Integration tests already exist against this code.
  • B-TEST CODE refers to the code that will be written. Unit and Integration Tests should be written against this code, as well.

This article is focused on the fact that additional tests (Unit and Integration) may need to be created to ensure these two sets of code work together properly with the existence of the other. Having said that ...

  • NEGATIVE-SCENARIOS should not be confused with Negative Testing which examines ways the code can fail. Negative Scenarios are the inverse of the tests for the alternate scenario (A-TEST CODE <-> B-TEST CODE).

Overview of Considered Scenarios

Examining the scenarios this way ...

Will the B-TEST move forward? General Test Coverage* Negative Scenario Coverage**
Little or no chance Few or no tests needed Few or no tests needed
Maybe Key areas Should have these
Very high chance Complete set of tests needed Complete set of tests needed

* These are tests written against the B-TEST code, testing the new functionality.
** These are tests that should be written as negative scenarios, testing for bleed-over of code between A-TEST and B-TEST scenarios. The tests for the alternate area should fail appropriately (B-TEST tests should include the INVERSE of A-TEST tests).

The Questions

  1. There should be test code for both the A-TEST and B-TEST scenarios. The question is how much test coverage should there be for the "new" code?
  2. Both scenarios need to be concerned about bleed-over from the other code. Are the negative versions of the other scenarios tests enough?
  3. Is there common code between the two scenarios that is covered appropriately (and therefore not included as negative test scenarios)?

Top comments (1)

chenxeed profile image
Albert Mulia Shintra

Thanks for this article!

As much as "having everything tested" is a clear practice, but some scenario shall be considered properly based on the use-case scenario, and unit testing AB-testing is also one of it.

I like the comparison table you've made to determine if the test is needed or not based on the probability of the "B-Test" scenario is going forward or not.

We can also have the consideration for the timing. If the "B-Test" scenario is likely going for production, but the test is not written yet during the development (because it's not written in TDD way), when do you think it's the best time to write the test then? I believe this is also one of the feedback given from the developers during review, but let me know if anyone think differently.