Who tests the tests?

Lucas Gabriel Sánchez — Fri, 20 Feb 2026 17:05:58 +0000

This post is based on a Gophercon talk by Daniela Petruzalek: Who tests the tests?

A little bit of history

In the beginning, we checked our code manually, running the application and trying different inputs: we called that manual testing. Then we discovered that we could write code to test application code to check if it's correct: we called that automatic testing (unit, integration, functional, etc.)

Now we are in the AI era where we write fewer tests and even less code, we need a way to swiftly check that the tests are correct and that they are testing the cases we expect in our application.

How can we know if code written by AI is correct?

You can use automatic tests in the same way we used to check code written by humans.

One option would be to let the AI write the application code and you write the tests, you can even use TDD where you, the human, write the tests and let the AI write the implementation after.

Another option is to let the AI write the application code and the tests, but then how can you be sure that those tests are testing the things you want or need? Reading and understanding the tests would be the best, but can we do that automatically?

Enter: mutation tests

Mutation testing is a way to check that the tests you have are testing the code the way you want by making small changes to the application code and checking that the tests fail as expected.

It's based on a concept known as a mutant: a version of your application with a small change.

How does it work?

The mutation testing cycle is this:

Create a mutant: change the application code by applying just one change
Run your test suite
Check how many mutants were killed

After running your tests, the ones that failed are said to have killed the mutant, those tests are testing something related to the code you changed and are, from the perspective of that change, good tests.

After many changes, if a test never failed it means that test didn't kill any mutant and is a weak test. You should probably delete that test or write a better one.

If a mutant is never killed, then that code is not being tested (no coverage) or is being tested poorly, you have an opportunity to write a test to check that piece of code if needed.

Do I have to make these changes manually?

You can do it manually, but there are some tools to aid you:

C#, TypeScript and Scala: Stryker
Go: go-gremlins
Java: pitest
Python: mutatest
Rust: mutants.rs

These tools provide a way to make those changes automatically and some of them run the tests for you.

Why do we need mutation testing?

Let's see a very simple example in Python:

def divide(a, b):
    if b == 0:
        raise ValueError("can't divide by 0")
    return a/b

A simple test suite we can have is:

import unittest
from divide import divide

class TestDivide(unittest.TestCase):
    def test_divide_error(self):
        with self.assertRaises(ValueError):
            divide(1, 0)

    def test_divide_success(self):
        self.assertEqual(1, divide(1, 1))

if __name__ == '__main__':
    unittest.main()

Run the tests and everything is fine:

$ python tests.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

Here we are testing both flows, when an error is raised and when we complete a successful operation, coverage is at 100% but those tests are not great, and here's why: let's change the implementation of divide from a/b to a*b:

def divide(a, b):
    if b == 0:
        raise ValueError("can't divide by 0")
    return a*b

Running the tests should fail, right? They don't, because 1*1 is still 1:

$ python tests.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

This is a problematic test:

def test_divide_success(self):
    self.assertEqual(1, divide(1, 1))

Even when you have 100% test coverage, that doesn't mean you have 100% test case coverage; some cases may be missing or not being tested correctly.

What we did here was a mutation: from a/b to a*b, that mutation gives us information about our tests that we didn't have before.

How to read the results of mutation testing?

When you run your test suite against mutated code, each test can do one of two things: kill the mutant (the test fails, so it detected the change) or survive (the test still passes, so it missed the change). After many mutation, you can summarize how often each test killed mutants, for example:

After 200 Mutations:
TestA: 140 Kills, 60 Survived
TestB: 200 Kills
TestC: 30 Kills, 170 Survived

How to interpret this:

TestA: is good test because on most mutations it was killed.
TestB: is your strongest test, it detects every mutation you ran.
TestC: is the weak one, it usually doesn't detect mutations.

When not to use mutation testing?

In some projects or languages, running mutation tests can take hours. For each mutant, the tool has to compile (if needed) and run the full test suite. If your test suite takes 10 minutes, each mutation will take at least that long, so keep this in mind.

Use this approach on small projects or ones where the "edit, compile, test" cycle is small.

Closing

So who tests the tests? In practice, mutation testing does that: by checking that your tests react when the code is deliberately broken.

Why you should learn Go?

Lucas Gabriel Sánchez — Tue, 06 Apr 2021 14:59:53 +0000

I can already hear you: Another programming language? WHY?!

There are already more than 650 programming languages.

Maybe you already know a few of them and can read and write one or two with great ease. Then if you can do X in Y programming language, why do you need to learn to do it in Go? To start answer that I will tell you a story, my story.

When I began my journey to become a programmer I wanted to know: What were the languages used in the wild?. I only knew Visual Basic, but didn't have any clue that almost nobody was using it in production. After a few months I discovered one programming language was dominating: Java. At that point I set the goal to learn Java and so I did.

I learned about object oriented programming and design patterns to understand Java. Also learned about classes and interfaces, about virtual methods and packages. The language was an excuse to learn a lot of theory about how to develop applications.

Since then I learned Pascal, Python, C, C++, C#, Erlang, ActionScript, JavaScript and Go. Every language had its own rules and quirks but a lot of them have the same principles and foundations. How many languages you know that use classes to define objects? How many languages use methods and the concept of "this" to refer to the object that receives the method call?

Later, a wise programmer told me: "The only programming languages worth learning are the ones that can teach you something new". Some languages taught me something new, but not all of them.

Some may say that learning Pascal or Erlang is not a good move in 2021, but they taught me things that I didn't know before.

From Erlang I learned the philosophy of its creators: "fail fast and fail loud". To have a resilient system, you should fail fast and loud so other parts of the system can do something about it.

From Python I learned that simple is better than complex. Explicit is better than implicit. Among other things

From C I learned that with great power comes great responsibility.

So, here are my 2 cents: Learn at least one programming language that's not your main one and use it. Expand you horizons, learn something new, learn Go. It will make you a better programmer.

Go is not the first language to implement some of its features but is the first to have them implemented well. Companies (and people) are using Go in production, in software that is changing the world. Some features present in Go that I didn't see in any other language until now are:

Concurrent primitives instead of threads or callbacks
Error values instead of exceptions and try/catch
Object orientation without classes nor inheritance
First class testing support
Deferred execution

Go also shares features with other programming languages, but implemented in different ways. Those ways let you see the world from a different perspective. A better perspective.

Those features are:

Interfaces: my favourite feature of Go.
It's open source
The Go 1 compatibility promise: Go 1 programs will always compile.
Type inference: in a statically typed language this is pure gold.
Compiled to machine code: it doesn't run in an interpreter or virtual machine, it runs fast and use less resources than others.
Fast compile times: you can compile several thousand of files into a single executable file in less than a second.
Garbage collected: you don't have to manage memory by hand.
Ease of use: tools such as "go build", "go test" and "go get" simplify the development cycle

If you learn Go you will acquire a different mindset. You will learn something new. You will see the world from a different perspective and that in itself can be a great thing for you and your career. No matter what language you use in 5 or 10 years from now.

DEV Community: Lucas Gabriel Sánchez