Audit AI-Generated Characterization Tests Before You Refactor

AI writes tests fast. That does not mean the tests protect your refactor. A passing test can still miss the behavior you are about to break. This pattern keeps showing up in legacy code. The solution is a short audit before you lock anything in.

Recent DEV discussions argue that AI turned every developer into a reviewer. Few people have a checklist for reviewing AI-generated tests. This guide gives you one. It focuses on characterization tests: tests that record existing behavior, not intended behavior. They are the safety net for refactoring messy code.

For this workflow, you can use MonkeyCode's free model access to draft initial tests. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode also offers a free server option. That gives you a disposable environment to run the suite. Any local setup or free CI works too.

The core problem

Characterization tests assert what the code does today. They do not assert what it should do. If the model guesses the expected value instead of running the code, the test is meaningless. Many generated tests use hard-coded values the model invented. Those tests pass for the wrong reason.

Here is a typical legacy function. It calculates shipping rates.

def shipping_rate(weight, priority=False):
    if weight <= 0:
        raise ValueError("weight must be positive")
    if priority:
        return 10.0
    return 4.0 if weight <= 1 else 6.0

Your task is to refactor it into a cleaner structure. You want a safety net first. A model might generate this test:

def test_shipping_rate_standard():
    assert shipping_rate(1) == 4.0
    assert shipping_rate(2) == 6.0

def test_shipping_rate_priority():
    assert shipping_rate(0.5, priority=True) == 10.0

Looks good. It is not enough. It misses the zero case, the error case, and the boundary between 1 and 1.0001. If your refactor accidentally lets weight zero through, the suite stays green.

The audit workflow

Here are five checks. Run them before you trust any generated characterization test.

1. Compare inputs to real usage

Look at production logs, saved queries, or user data. Choose inputs that actually happened. Synthetic inputs are fine, but they must cover the same branches. If the model wrote tests for values you never see, replace them.

2. Confirm the expected value was read, not invented

Read each assertion. Did the model copy the value from the source code? Or did it predict the output? In the example above, shipping_rate(1) == 4.0 could be a guess. The only way to know is to run the code. Never leave a test you did not personally execute at least once.

3. Cover the boundaries

List every conditional boundary. For shipping_rate, boundaries are weight <= 0, weight <= 1, and the priority flag. Add tests for weight = 0, weight = 1, weight = 1.0001, priority = True. For each, record the current output. That becomes your golden master.

4. Run a mutation test

Deliberately change one line. For example, change weight <= 1 to weight < 1. Run the suite. The test that pins that boundary should fail. If no test fails, you have a hole.

# after applying the mutation manually
pytest test_shipping_rate.py

A red result is good. It proves the test can catch the regression you care about. Then revert the mutation.

5. Keep the suite fast and isolated

Your characterization tests will run on every refactor commit. They must be fast. Generate fixtures inside each test; do not rely on shared state. Use unique module names. If the code touches external services, replace them with fakes.

The free server option from MonkeyCode is one way to get that isolation. You can also use a container or a temporary VM. Pick whatever is reproducible.

A reproducible audit command

Here is a command that checks whether your generated tests actually cover the branches. It uses the coverage tool.

coverage run -m pytest test_shipping_rate.py
coverage report -m --branch

The output shows uncovered lines and branches. For the example above, you should see missing branches around weight <= 0 if you skipped the zero case. Add tests until the report is clean.

pip install pytest coverage
coverage run -m pytest test_shipping_rate.py
coverage report -m --branch

Do not declare your refactor safe with a known missing branch. A green suite with low coverage is still a gamble.

When not to use this approach

Characterization tests are not for every project. Skip them when the code will be deleted soon. Skip them when you already have a full suite with meaningful assertions. Skip them when the behavior is obviously wrong and you have permission to change it. In those cases, write intent-testing tests instead.

Never let generated tests replace human judgment. The model is a drafting tool. You are the reviewer. Your job is to decide which behavior deserves protection.

The bottom line

Generated characterization tests reduce the effort of starting a refactor. They do not remove the responsibility of auditing them. Use real inputs, verify expected values, cover boundaries, and run a mutation test. That is a small price for a safe refactor.

Start with one function. Generate your candidates. Audit them hard. Then refactor with confidence.