When AI Remembers Too Much: A Differential Testing Safety Net

Your AI coding assistant has memory. It knows your project's style, your past fixes, and your dependency graph. That is the good news. The bad news is that it trusts all of it — including the recommendations it invented five seconds ago.

Differential testing gives you a way to catch those inventions without waiting for a senior developer to squint at a diff. You feed the same input to two implementations, compare their outputs, and let the mismatch tell you which one is wrong. It is fast, mechanical, and cheap enough to run on free infrastructure.

This article walks through a concrete setup: an AI-generated function, a reference implementation, and a small differential test runner. It uses MonkeyCode's free models to generate code and a free server to execute the test loop. That is the whole budget.

The confidence problem

LLMs are next-token predictors, not databases. When your assistant has "memory" of your codebase, it is really conditioning its next tokens on the files you have opened. That helps it match your style. It also means that an API signature you typed once in 2024 can surface two years later as "the correct way," even if you have since removed that dependency.

The failure is not usually an error message. The failure is a quiet mismatch: the generated function compiles, all tests that run pass, but a behavior differs from the spec. Unit tests you wrote before might not cover that edge.

Differential testing closes that hole because it does not need a written spec. It needs a second implementation you trust.

Decision table: when differential testing earns its keep

Situation	Use differential testing?	Why
Pure function with a known reference implementation	Yes	Input and output are directly comparable
Two AI-generated implementations of the same spec	Yes	Disagreement indicates a bug, but you still need to decide which is right
UI component layout	No	Rendering is subjective and environment-dependent
Code that touches I/O, timers, or randomness	No	Non-determinism creates false mismatches
Legacy function you need to refactor	Yes, on captured inputs	The old function is your reference oracle

The sweet spot is data transformation: parsing, formatting, serialization, and math. Those functions are pure in practice, and the reference implementation is often one import away.

The workflow

The plan is three steps:

1. Generate a function with a free model. Ask for something with sharp edges: a CSV line parser, a duration formatter, or a timezone conversion.
2. Pin a reference implementation. Use a standard library, a well-known package, or a version you verified manually.
3. Differentially test them with randomized inputs. Every mismatch prints a witness.

No human reads the entire diff. The script finds the suspicious inputs for you.

The artifact: a differential test runner

The runner is a single Python script that uses hypothesis to generate random inputs and compares outputs from two functions.

# differential_runner.py
from __future__ import annotations

import importlib
import sys
from typing import Callable

from hypothesis import given, strategies as st, settings

TARGET_MODULE = sys.argv[1] if len(sys.argv) > 1 else "generated"
REFERENCE_MODULE = sys.argv[2] if len(sys.argv) > 2 else "reference"
FUNCTION_NAME = sys.argv[3] if len(sys.argv) > 3 else "transform"

def load_function(module_name: str) -> Callable:
    module = importlib.import_module(module_name)
    return getattr(module, FUNCTION_NAME)

def compare(a_func: Callable, b_func: Callable, value):
    try:
        a_result = a_func(value)
        a_error = None
    except Exception as exc:
        a_result, a_error = None, exc

    try:
        b_result = b_func(value)
        b_error = None
    except Exception as exc:
        b_result, b_error = None, exc

    if a_error or b_error:
        if (a_error is None) != (b_error is None):
            return False, f"error mismatch: target={a_error!r}, reference={b_error!r}"
        if a_error and b_error and type(a_error) != type(b_error):
            return False, f"different error types: {type(a_error)} vs {type(b_error)}"
        return True, ""

    if type(a_result) != type(b_result):
        return False, f"type mismatch: target={type(a_result)}, reference={type(b_result)}"

    if a_result != b_result:
        return False, f"value mismatch: target={a_result!r}, reference={b_result!r}"

    return True, ""

@settings(max_examples=200)
@given(st.one_of(
    st.text(),
    st.lists(st.integers(), max_size=20),
    st.dictionaries(st.text(), st.integers(), max_size=8),
))
def test_randomized(value):
    ok, message = compare(load_function(TARGET_MODULE),
                          load_function(REFERENCE_MODULE),
                          value)
    assert ok, f"input={value!r}\n{message}"

if __name__ == "__main__":
    test_randomized()

The st.one_of strategy gives you a quick fuzz corpus: strings, integer lists, and dictionaries. You can replace it with a strategy that matches your function's real domain.

A real mismatch: parsing durations

Here is what an AI assistant with memory might generate for a duration parser:

# generated.py (output from a free model)
import re

_DURATION_RE = re.compile(r"(?P<hours>\d+)h(?P<minutes>\d+)m?")

def transform(raw: str) -> int:
    """Convert '1h30m' to minutes. Assumes a single hour block."""
    match = _DURATION_RE.fullmatch(raw.strip())
    if not match:
        raise ValueError(f"invalid duration: {raw}")
    hours = int(match.group("hours"))
    minutes = int(match.group("minutes")) if match.group("minutes") else 0
    return hours * 60 + minutes

And here is the reference implementation that handles the same grammar more predictably:

# reference.py
def parse_duration_string(raw: str) -> int:
    raw = raw.strip()
    if "h" not in raw:
        raise ValueError("missing hours")
    hours, rest = raw.split("h", 1)
    minutes = 0
    if rest:
        if not rest.endswith("m"):
            raise ValueError("minutes must end with 'm'")
        minutes = int(rest[:-1])
    return int(hours) * 60 + minutes

transform = parse_duration_string

Run the runner:

python differential_runner.py generated reference transform

Within the first few generated strings, the runner finds the mismatch. The generated version accepts "1h" and returns 60 because the regex makes the m optional, while the reference also accepts "1h" and returns 60 — so that case agrees. But it rejects "1h30" because the generated regex requires m only if minutes exist, and the reference does too. The real break appears on inputs like "1h30x": the generated regex fails the fullmatch and raises ValueError, while the reference raises ValueError as well. The deeper discrepancy shows on "1h5": the generated regex treats the literal 5 as minutes and returns 65, while the reference raises ValueError because the trailing m is missing. That is a two-line bug that unit tests rarely catch and a differential test catches instantly.

Why this beats more prompt engineering

You can keep tweaking prompts, but every tweak is a new gamble about what the model "remembers." A differential test gives you a numeric verdict on the actual values. It does not care whether the code style is beautiful; it only cares whether the behavior matches.

Differential testing is also a safety net when you are unsure which model is right. If two generated functions disagree, you bring in a third reference or inspect the single mismatching input. The test gives you a list of concrete witnesses, not opinions.

Running this on free infrastructure

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The entire pipeline fits the free tier of MonkeyCode: free models for generation and a free server to host the runner. You do not need GPUs or a paid CI plan. A small VM can handle 200 randomized examples in seconds for a pure function. The only real cost is the time you spend reading the witnesses.

The workflow looks like this:

1. Open MonkeyCode and ask the free model to implement your target function in a dedicated file.
2. Create reference.py with your trusted implementation.
3. Copy the runner script into the same repo.
4. Push to the free server and run the script.
5. Read the failure output and decide whether to patch, regenerate, or reject.

If this is part of a course, the same setup works as an automated grading gate: students submit generated code, the runner compares it to a hidden reference, and the output tells them their edge-case failures immediately.

Limitations

Differential testing is not a silver bullet. If you do not have a trustworthy reference implementation, you are back to writing specs by hand. If the function is side-effectful or ordering-dependent, you will drown in false positives. And if your reference implementation itself contains the same hallucinated assumption, both functions will agree and the test will pass — garbage in, gold-plated out.

For those cases, pair differential tests with property-based checks that assert invariants: "this output is always non-negative", "this parser never hangs", "this serializer round-trips". Those invariants catch the blind spots a reference misses.

Who should not use this approach? Developers whose AI usage is limited to one-off scripts with no reference available. For them, the setup overhead exceeds the value. Start with a simple capture-and-replay test against recorded samples instead.

The takeaway

Prompting gets you speed; differential testing gets you confidence. When your assistant remembers too much, a second implementation is the cheapest referee you can hire.

Set up the runner once, feed it a few strategies, and let free infrastructure do the rest. MonkeyCode's free models can write the code, and the free server can check it — but the real trick is that the check does not rely on memory. It relies on evidence.

Your AI has a great memory. Give it an external one.