Now, this seems like something that should be pretty straight forward ... testing the test and all that; what-what. But there are a few underlying considerations that make how to Unit Testing A/B Test Code a fairly remarkable question.
While writing this article about testing test code, the language started to get a bit wonky.
Here are the definitions I will use moving forward ...
- A-TEST CODE refers to the code that should already exist when it is decided that and A/B Test is needed. The expectation should be that Unit and Integration tests already exist against this code.
- B-TEST CODE refers to the code that will be written. Unit and Integration Tests should be written against this code, as well.
This article is focused on the fact that additional tests (Unit and Integration) may need to be created to ensure these two sets of code work together properly with the existence of the other. Having said that ...
- NEGATIVE-SCENARIOS should not be confused with Negative Testing which examines ways the code can fail. Negative Scenarios are the inverse of the tests for the alternate scenario (A-TEST CODE <-> B-TEST CODE).
Examining the scenarios this way ...
|Will the B-TEST move forward?||General Test Coverage*||Negative Scenario Coverage**|
|Little or no chance||Few or no tests needed||Few or no tests needed|
|Maybe||Key areas||Should have these|
|Very high chance||Complete set of tests needed||Complete set of tests needed|
* These are tests written against the B-TEST code, testing the new functionality.
** These are tests that should be written as negative scenarios, testing for bleed-over of code between A-TEST and B-TEST scenarios. The tests for the alternate area should fail appropriately (B-TEST tests should include the INVERSE of A-TEST tests).
- There should be test code for both the A-TEST and B-TEST scenarios. The question is how much test coverage should there be for the "new" code?
- Both scenarios need to be concerned about bleed-over from the other code. Are the negative versions of the other scenarios tests enough?
- Is there common code between the two scenarios that is covered appropriately (and therefore not included as negative test scenarios)?