Using Control and Test Groups in Tests

Computer Science degrees at any university will focus a lot on mathematically proving things; after all, it is a science. But then, in the real world, most jobs don't require you to come up with new complex algorithms and mathematically prove them. Instead, most of the time, you only need to write tests to prove that your code works. The problem is that sometimes, these tests are not rigorous enough.

Well designed experiments

Consider a study where you need to check if a pill cures an illness. As a simple experiment, you could use two groups of patients; some receive a placebo (control group), and the others receive the real pill (test group). They don't know which pill they took, and the only difference between the two groups is the pill. If the pill works, there would be a considerable difference in the number of patients recovered in the test group vs the control group. However, if the pill doesn't work, then the outcome in both groups will be pretty much the same, or people in the test group could be even sickier 🫣. We need a control group because, otherwise, how would you know that the people who got the pill recovered because of it and nothing else?

Writing tests as experiments

Consider, for example, a Website where there is an admin url that only users with admin permissions can access, and anyone else would see a 404.

You could quickly create a test like this:

def test_admin_access():
    regular_user = make_user()
    client = HTTPClient()
    response = client.get("http://localhost:1234/admin")
    assert response.status_code == 404

As per the pills and patient example, how do you known the 404 code is not due to something else? In fact, the user is not logged in, which could be causing a 404. And what if the URL is wrong? Wouldn't that also cause a 404? So, this test can pass for the wrong reasons, and the admin endpoint could inadvertently allow access to regular users.

Well, let's change it a little bit:

def test_regular_user_cannot_access():
    regular_user = make_user()

    client = HTTPClient()
    client.login(regular_user)

    response = client.get("http://localhost:1234/admn")
    assert response.status_code == 404

def test_admin_user_can_access():
    admin_user = make_user(admin=True)

    client = HTTPClient()
    client.login(regular_user)

    response = client.get("http://localhost:1234/admin")
    assert response.status_code == 200

WRONG!!!

While it looks like we added a control group at first sight, we are not performing the same experiment in both. Note the typo in the URL for the first test; the request will return 404 because the URL is wrong and the test will "pass".

Fixing the typo is not a real fix. The right way to fix this is to separate the experiment into its own function and call it for each user type.

def get_admin_page(user):  # experiment
    client = HTTPClient()
    if user is not None:
        client.login(user)
    return client.get("http://localhost:1234/admin")

def test_admin_access():
    regular_user = make_user()
    admin_user = make_user(admin=True)

    # Test
    no_user_response = get_admin_page(None)
    regular_user_response = get_admin_page(regular_user)
    # Control
    admin_user_response = get_admin_page(admin_user)

    # Test
    assert no_user_response.status_code == 404
    assert regular_user_response.status_code == 404
    # Control
    assert admin_user_response.status_code == 200

Bonus: we are also testing that if the user hasn't logged in, it can reuse the same control group.

While I have added them under the same test function, it doesn't matter if you break it into multiple tests as long as they all use get_admin_page.

We were able to test it as if it was an experiment because the users are objects, where we change an independent variable, i.e. is_admin or is_logged_in, and we check that the experiment (get_admin_page) returns the expected result for each.

When it doesn't apply

There are some cases where you need to resort to other methods. Consider the following function:

def get_percentage(portion, total):
    portion/total * 100

You could try and come up with a control and test group, but under which criteria are you separating the groups? Both portion and total are independent variables, but we are not performing an experiment over an object. A different way could be proving it mathematically, though it is a simple formula. So, perhaps the best way to ensure that you didn't type 101 instead of 100 is to write a simple test that checks that the result is as expected for certain inputs.