Same Input, Different Answer? A Differential Harness for AI Code Fixes

A patch that compiles is not a patch that behaves. I keep rediscovering this truth every time an AI suggests a one-line change to a parser or a state machine, and the tests pass while some obscure edge case silently changes meaning.

Differential testing is the cheapest way to expose that mismatch. Instead of asking "does the new code produce the expected output?", you ask "does the new code produce the same output as the old code for every input you can throw at it?". When the old code is the specification, any disagreement becomes a suspect.

That idea is simple. The practical challenge is running it at scale: you need a model that can generate candidate fixes, and a server that can execute hundreds of test cases without melting your laptop. That is exactly where MonkeyCode's free model access and free server option come into play, and I will show how to build a minimal differential harness that runs entirely inside that free tier.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The setup: a function that keeps losing its invariants

Let's take a deliberately silly but realistic example: a function that compresses a run-length encoded string and then decompresses it back.

def rle_roundtrip(plain: str) -> str:
    if not plain:
        return ""
    encoded = []
    count = 1
    for i in range(1, len(plain)):
        if plain[i] == plain[i-1]:
            count += 1
        else:
            encoded.append(plain[i-1] + str(count))
            count = 1
    encoded.append(plain[-1] + str(count))
    return "".join(encoded)

The function looks correct at first glance, but it loses information when a character repeats more than nine times. The string "aaaaaaaaaa" (ten a's) becomes "a10", which decompresses as one a followed by the digits 1 and 0. A human reviewer might catch that; an AI model might not.

Now imagine you ask a coding model to fix this bug. It returns a candidate patch that changes the encoding scheme to include a separator. Your golden tests pass. But what if the patch also changes the output for inputs that were already working? Differential testing will detect that instantly.

Building the differential harness

The idea is to run the original function and the patched version against the same stream of random inputs, and compare their outputs. If they differ, the patch has changed behavior in some way — possibly the fix, possibly a regression.

import random
import string

def random_string(max_len: int = 30) -> str:
    length = random.randint(0, max_len)
    return "".join(random.choice(string.ascii_lowercase) for _ in range(length))

def differential_test(original, patched, trials: int = 1000) -> list:
    mismatches = []
    for _ in range(trials):
        data = random_string()
        try:
            a = original(data)
            b = patched(data)
        except Exception as exc_a:
            try:
                patched(data)
            except Exception:
                continue  # both failed, that's fine
            mismatches.append((data, "exception only in original", exc_a))
            continue
        try:
            patched(data)
        except Exception as exc_b:
            mismatches.append((data, "exception only in patched", exc_b))
            continue
        if a != b:
            mismatches.append((data, a, b))
    return mismatches

This is deliberately naive. It does not compare error types, it does not respect invariants, and it treats any difference as flag-worthy. That is the point: differential testing is a broad net, not a precise filter.

Where the free server enters the picture

Running 1,000 trials on a local machine is fast for this tiny function. Real code is not that tiny. You might be testing a JSON normalizer, a URL parser, or a cache eviction policy — each trial could load libraries, allocate memory, and hit filesystem caches. That is where the compute burden grows, and why a disposable sandbox helps.

MonkeyCode's free server option gives you a remote execution slot without you having to provision one. You can POST a candidate patch and a list of inputs, get back a result matrix, and tear it down. The workflow looks like this:

1. Generate a candidate patch using MonkeyCode's free model access.
2. Send the patch plus a small random seed batch to the free server.
3. Run the differential harness in that remote environment.
4. Collect the mismatches and send them back as feedback to the model for a second attempt.

The model and the server are separate resources. The free model gives you as many candidate generations as you need to explore different approaches; the free server gives you the isolated execution environment to validate them without contaminating your dev machine with half-finished dependency versions.

A decision table for choosing your validation level

Different situations call for different levels of scrutiny. Here is the table I use before deciding how much effort to spend on evaluating an AI patch:

Situation	Compile/build check	Golden cases	Differential test	Property test
One-line constant change	Yes	Yes	Maybe	No
Algorithm replacement	Yes	Yes	Yes	Yes
Refactoring with no behavior change	Yes	Yes	Yes	No
Adding a new feature	Yes	Yes	No	No
Fixing a race condition	Yes	No	No	Yes

Differential testing shines when behavior should stay identical — a refactor, an optimization, a bug fix. It fails when the expected behavior changes deliberately, which is why feature additions usually skip it.

The good news is that you can run all of this within a free account. I have used MonkeyCode's free tier to run this exact harness against several small Python projects, and the only cost is the time you spend interpreting the mismatches.

Limitations you need to accept

Differential testing assumes the original implementation is the source of truth. If the original has a bug that the patch is supposed to fix, any input that trips that bug will show up as a mismatch — even though the patch is technically correct. You need to triage those carefully, or mark those inputs as expected differences.

Non-deterministic functions break the comparison. Any code that involves randomness, time, or asynchronous ordering will produce flaky mismatches unless you inject fixed seeds or freeze the clock.

The free server is not a massive parallel cluster. You will queue dozens of patches, not thousands. For heavy fuzzing campaigns, you should still fall back to your own CI.

Finally, the free model access has its own limits on rate and context, which are documented on the MonkeyCode dashboard. I did not measure those limits, and I recommend checking them before promising a 10,000-case run to your team.

The workflow that finally made sense for me

After a few weeks of using this differential approach, my routine settled into three steps.

First, I ask the model for a patch using MonkeyCode's free model, without giving it a golden test. Second, I run the differential harness with a fixed random seed so the mismatches are reproducible. Third, I copy the first mismatch back into the prompt and ask the model to explain what changed.

That last step is the most valuable part. The model often turns the mismatch into a crisp explanation: "Oh, I inverted the condition to handle empty strings, but I forgot that None is also possible." Sometimes it even proposes a better fix on its own.

The free server makes this loop painless because I do not need to manage a Python environment for every trial. I paste the patch and the seed, and the result comes back as a small JSON report.

Should you adopt this today?

If you already have a local test matrix that covers the critical paths, differential testing might feel redundant. If you are reviewing AI suggestions for a small utility that your CI already tests, the added harness is probably overkill.

But if you are merging AI-generated patches into a codebase where subtle behavior changes have caused production incidents before, I would start with a differential harness before anything else. It is cheap to write, runs comfortably on a free server, and converts the vague feeling of "this patch might break something" into an actual list of violating inputs.

That list is what makes the review conversation productive. You are no longer arguing about trust or vibes; you are arguing about concrete cases where the old code and the new code disagreed.

And when the disagreement disappears, you can merge with a little more confidence — until the next patch arrives.

---

If you want to try this differential harness with MonkeyCode, the free model access and free server option are available today. Feed them a buggy function, generate two or three candidate fixes, and watch how many mismatches disappear.