Why Generated Tests Fail: A Field Guide to 5 Failure Modes

Generated tests fail in predictable ways. Once you see the pattern, you can fix it. This guide covers five failure modes. Each has a code example and a fix. The goal is simple: turn a pile of broken drafts into a usable test baseline.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why this matters

Model-generated tests rarely pass on the first run. That is not the model's fault. Tests need context. The model only sees the source you give it. The failure modes are finite. Learn them, and you can batch-fix most of your generated suite.

This guide assumes you have a generation pipeline. I use MonkeyCode's free model access to generate tests. The free server runs pytest. The free tier includes a 10M token allowance. One generation costs roughly 800 tokens. You can run thousands of iterations.

Your pipeline probably looks like this:

import json
import os
import urllib.request

def generate_test(func_name, source):
    prompt = f"""Write one pytest test for {func_name}.
Source:
{source}
Return only Python code."""
    payload = {
        'model': os.environ['MONKEYCODE_MODEL'],
        'messages': [{'role': 'user', 'content': prompt}],
    }
    req = urllib.request.Request(
        os.environ['MONKEYCODE_BASE_URL'],
        data=json.dumps(payload).encode(),
        headers={
            'Authorization': f"Bearer {os.environ['MONKEYCODE_API_KEY']}",
            'Content-Type': 'application/json',
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        data = json.load(resp)
    return data['choices'][0]['message']['content'].strip()

Failure Mode 1: The Overly Strict Assertion

Models love exact matches. Real functions return timestamps, random IDs, or floats. Exact assertions fail immediately.

A typical model output:

def test_format_report():
    assert format_report({"name": "Alice"}) == "Report: Alice at 2026-08-22 10:00:00"

The problem: the timestamp is generated at runtime. The fix: assert substrings.

def test_format_report():
    result = format_report({"name": "Alice"})
    assert "Alice" in result
    assert result.startswith("Report:")

Detection signal: the failure message shows a string mismatch, but only the dynamic part differs.

Failure Mode 2: The Missing Environment

The model does not know your code needs config files, env vars, or a database. It generates a direct call. The function crashes because the environment is absent.

A typical model output:

def test_load_config():
    config = load_config()
    assert config["timeout"] == 30

The problem: load_config() reads config.yaml, but the test directory has no such file. The fix: add a fixture.

def test_load_config(tmp_path, monkeypatch):
    (tmp_path / "config.yaml").write_text("timeout: 30\n")
    monkeypatch.chdir(tmp_path)
    assert load_config()["timeout"] == 30

Detection signal: FileNotFoundError, KeyError, or connection errors.

Failure Mode 3: The Wrong Argument Guess

The model sees process(data, mode) and guesses data is a string. In reality, data is a list of dicts. The test dies at the entry point.

A typical model output:

def test_process():
    assert process("hello", "fast") == ["hello"]

The problem: process expects data to be list[dict]. The fix: infer types from the source before generating.

def test_process():
    data = [{"id": 1, "text": "hello"}]
    result = process(data, "fast")
    assert result[0]["id"] == 1

Detection signal: TypeError or AttributeError on the first line of the test. Give the model more source context to reduce this mode.

Failure Mode 4: The Hidden Global State

Module-level variables, caches, or singletons. The model cannot see them. It assumes the function is pure. The test passes alone and fails in a full run.

A typical model output:

def test_get_counter():
    assert get_counter() == 0

The problem: get_counter() reads module-level _counter. The first call changes it to 1. The fix: reset state.

def test_get_counter(monkeypatch):
    import mymodule
    monkeypatch.setattr(mymodule, "_counter", 0)
    assert mymodule.get_counter() == 0

Detection signal: the test passes in isolation and fails with the full suite. This is the sneakiest mode.

Failure Mode 5: The Empty Assertion

The most dangerous one. The test passes but verifies nothing. The model generates assert result is not None. A function returning None sails through.

A typical model output:

def test_parse():
    result = parse("a=1")
    assert result is not None

The problem: an empty dict also passes. The fix: assert concrete values.

def test_parse():
    result = parse("a=1")
    assert result == {"a": "1"}

Detection signal: coverage rises, but mutation testing kills everything. That is false confidence.

The classification workflow

Do not review generated tests one by one. Run them first. Then classify.

pytest test_generated.py --tb=short --no-header -q

Map each failure to a mode. Each mode has a standard fix. Apply them in bulk.

Failure mode	Signal	Fix
Overly strict assertion	String mismatch	Use substring assertions
Missing environment	FileNotFoundError	Add a fixture
Wrong argument guess	TypeError	Correct the arguments
Hidden global state	Passes alone, fails together	Reset state
Empty assertion	Passes but verifies nothing	Strengthen the assertion

What to generate on

Not every function deserves generated tests. Use this decision table.

Function trait	Generate?	Why
Pure function, simple args	Yes	The model guesses correctly
Clear input/output types	Yes	Assertions are easy to write
Depends on IO or network	No	Too many environment traps
Depends on global state	No	Tests become flaky
Security-critical logic	No	Needs human review

Limitations

This guide covers common failure modes. Your codebase may produce new ones. The table is not exhaustive.

Generated tests are drafts. They do not replace hand-written tests. They give you a starting point.

The free token allowance is generous but not infinite. Watch your usage dashboard.

The honest bottom line

Generated test failures are predictable. Predictable means fixable. Fixable means you can turn AI drafts into a real baseline.

Start with one pure-function file. Run the pipeline. Classify failures with this guide. Most fixes take a few lines. Your future self will thank you.