When a small refactor touches a hot path, the first question is often not 'is the new code clean?' but 'did it change behavior?' Most teams try to answer that with a code review plus the existing test suite. The weakness is well known: the suite tends to cover the happy path, and a human reviewer is reading a diff, not systematically exploring inputs. That is when a free model starts to look like a shortcut: ask it to rewrite the function. The problem is that you then have a new implementation whose edge cases you must audit from scratch. A safer division of labor is to let the model generate candidate inputs, while the pre-refactor code supplies the oracle for what the outputs should be.
To make that concrete, I will use MonkeyCode's free model access for the input-generation pass. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode also mentions a free server option; I treat that only as a possible place to run the comparison, and I am not assuming specific limits or hardware beyond that.
Why not just ask the model to make the change?
If the model rewrites the function, you have doubled the review surface. The diff may be small, but the semantic risk is now spread across every input the model considered while rewriting. A review of the diff only tells you what changed, not what stopped working. To find what stopped working, you need a set of inputs that are likely to expose behavior differences. That is a test-generation problem, not a code-generation problem.
Free models are useful here because they can produce a broad, messy list of edge cases from a short prompt. Their output does not have to be authoritative. You do not need the model to know the correct answer; you already have the correct answer, frozen in the old implementation. You only need the model to suggest inputs that your hand-written seeds missed.
A differential oracle in one paragraph
Take a function that is pure enough to call with simple values: a string normalizer, a date parse helper, a URL builder, a small state machine transition. Keep the old and new versions side by side. Run both with the same inputs. If both produce the same normalized output, the refactor is consistent with old behavior for that input. If they differ, record the input as a review item. The model's job is to expand the input list; the old code remains the oracle.
This does not prove the refactor is correct. It proves only that the refactor preserved the legacy behavior on the tested set. That is a much smaller and more honest claim, but it catches the failure mode most teams actually fear after a cleanup: a silent change on an unusual input.
A minimal harness
The harness below is intentionally plain Python. I am describing the shape of the workflow rather than shipping a tested library, so adapt the serialization and exception handling to your codebase.
from dataclasses import dataclass
from typing import Callable, Iterable
@dataclass
class InputSpec:
name: str
parser: Callable[[str], object]
seeds: Iterable[object]
def normalize(value):
if isinstance(value, float):
return f'{value:.15g}'
return repr(value)
def run_differential(old_fn, new_fn, spec, variants):
mismatches = []
seen = set()
def compare(label, value):
try:
old_out = old_fn(value)
except Exception as old_exc:
try:
new_out = new_fn(value)
except Exception as new_exc:
# Both raised. A mismatch only if the failure class changed.
if type(old_exc).__name__ != type(new_exc).__name__:
mismatches.append((label, f'old {type(old_exc).__name__}', f'new {type(new_exc).__name__}'))
else:
mismatches.append((label, f'old raised {type(old_exc).__name__}', 'new returned'))
return
try:
new_out = new_fn(value)
except Exception as new_exc:
mismatches.append((label, 'old returned', f'new raised {type(new_exc).__name__}'))
return
if normalize(old_out) != normalize(new_out):
mismatches.append((label, normalize(old_out), normalize(new_out)))
for seed in spec.seeds:
key = repr(seed)
if key not in seen:
seen.add(key)
compare(key, seed)
for text in variants:
text = text.strip()
key = f'text:{text}'
if not text or key in seen:
continue
seen.add(key)
try:
value = spec.parser(text)
except Exception:
# The model produced something the parser cannot accept.
continue
compare(text, value)
return mismatches
Use repr as a cheap serializer for most values, and replace it with something domain-specific when object identity or precision matters. For floats, compare fixed-precision strings instead of raw equality so tiny numeric drift does not flood the report. For inputs that the model returns as text, add a parser to turn a line into the typed value your function expects. Any line that cannot be parsed is simply skipped; that filtering is part of the safety boundary between the model and your code.
Prompt for inputs, not for code
A short prompt gives the model room to be useful without giving it permission to change the behavior. For a surname normalizer, the prompt can be:
You are generating test inputs for a function that normalizes a surname for a billing address.
Do not write code. Return one candidate input per line. Include:
- accented Latin scripts
- apostrophes, hyphens, spaces, and tabs
- leading and trailing spaces
- very long strings over 200 characters
- non-Latin scripts
- punctuation-only strings
- numeric strings
- strings that look like JSON, HTML, or SQL
Return only the lines. No bullets, no explanations.
Collect the result, strip whitespace, remove duplicates, and throw away lines that look like prose. The goal is volume and strangeness, not completeness. If the model drifts and returns a sentence, your parser or deduplication should drop it before it reaches the function.
What to do with the mismatches
Treat a mismatch as a lead, not a bug report. If the new code changes a normalized name, you may have found a real regression. If the old code raised and the new code returns a value, that may be an intentional relaxation of the input contract—or an accidental one. If both paths returned values that normalize differently, read the input before trusting the new code.
Keep the mismatch list small enough for human review. Start with a few dozen seeds and a few hundred generated variants. A report with hundreds of noisy differences is less useful than twenty input/output pairs that are easy to verify.
Limitations
This workflow only checks equivalence with the old implementation. It will not catch a bug that existed before the refactor, and it will resist any deliberate behavior change unless you update the oracle. Purity is a real constraint: functions with randomness, wall-clock time, filesystem access, or network calls need wrappers, and those wrappers can hide differences. Free model output is also not deterministic direction: it can repeat the prompt's examples, imitate training data, or generate irrelevant text, so filtering and deduplication are not optional. Finally, a free server is not a substitute for a controlled runner; verify what can and cannot be sent there before putting sensitive inputs or credentials near the model endpoint.
When this is a poor fit
Skip this if the refactor is intentionally changing a public contract, if the function has heavy side effects, or if you already have a property-based testing tool such as Hypothesis or a grammar fuzzer that can explore the same input space with better coverage. The free model is most useful when it brings domain-shaped guesses—names, dates, API parameters—where a pure grammar would need a lot of hand modeling before it catches anything real.
The point is not to get more AI in your pipeline. The point is to use a free model where its weakness matters least: suggesting strange inputs. If you have such an endpoint and a pure function that has accumulated edge-case assumptions, try the smallest version first. Generate fifty candidate inputs before a rewrite, run both implementations, and see what surfaces. You do not need the model to be correct; you only need it to be curious about the inputs you forgot.
Top comments (0)