A legacy parser, zero tests, and a refactor that had to ship by Friday. That's the setup. The twist is how I got the safety net: I didn't ask the model to write code. I asked it to define correctness.
Here's the story.
The background
The module was a log parser written three years ago by someone who no longer works here. It handled four formats, each with its own quirks. It worked. Nobody knew why it worked. And I had to restructure it to support a fifth format.
No tests. None. The original author believed in "careful coding." The rest of us believed in not touching that file.
My goal was simple: before changing anything, I needed a behavioral safety net. Something that would scream if the refactored parser produced different output than the original.
The implementation
The idea was to use a free model as an oracle — not to generate code, but to generate invariants. Properties that any correct implementation must satisfy. Then I'd encode those properties as Hypothesis tests and run them against both the old and new implementations.
The whole pipeline ran on MonkeyCode's free model access — the prompts, the invariant extraction, the test runs. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The workflow had three steps.
First, I pasted the parser's source into a prompt and asked one question: "What invariants must this function's output satisfy, regardless of input format?" No code. Just properties.
The model came back with a list. Most were obvious — the output must contain a timestamp, the level must be one of four values. But a few surprised me:
- The parsed timestamp must be monotonically non-decreasing when inputs are in file order.
- The message field must never be empty, even when the raw log line has no message.
- The line number in the output must match the physical line number in the input.
That third one was the gem. I hadn't thought about line numbers at all.
Second, I turned those invariants into Hypothesis tests:
from hypothesis import given, strategies as st
from hypothesis import settings
@settings(max_examples=1000)
@given(st.text(min_size=1, max_size=500))
def test_timestamp_is_iso_or_none(raw_line):
parsed = parse_line(raw_line)
if parsed is None:
return
assert parsed["timestamp"].startswith("2026-") or parsed["timestamp"] == ""
@given(st.lists(st.text(min_size=1), min_size=1, max_size=100))
def test_line_numbers_match_physical_position(lines):
parsed = parse_log("\n".join(lines))
for i, entry in enumerate(parsed):
assert entry["line_number"] == i + 1
The first test was weak — it just checked the timestamp format. The second was the real one. It caught a bug immediately.
Third, I ran the tests against the original parser to establish a baseline. Then I ran them against my refactored version.
The results
The line-number test failed on the refactor. Not because the refactor broke line numbers — because the original parser had a bug that the refactor had accidentally fixed.
The original code skipped blank lines but didn't increment the line counter. So after a blank line, every subsequent entry had a line number that was off by one. My refactor used enumerate, which counted every line, blank or not. The outputs differed.
Here's the thing: without the oracle, I would have "fixed" the line numbers and shipped a behavior change that looked like an improvement. The test forced me to decide explicitly: preserve the bug, or fix it and tell everyone.
I chose to preserve it. Not because the bug was good, but because the parser's output feeds a downstream system that keys on line numbers. Changing them silently would have corrupted that system's state.
The model's invariant — "line numbers must match physical position" — was wrong for this codebase. But it was usefully wrong. It surfaced a decision I would have made by accident instead of by choice.
The test suite ran on MonkeyCode's free server option, on an hourly cron. The token cost was trivial — a few hundred thousand tokens for all the prompt iterations and test runs, out of the 10 million free allocation.
The limitations
Let me be clear about what this approach doesn't do.
It doesn't verify semantic correctness. The invariants I got were structural — types, formats, ordering. The model never saw the original requirements document, so it couldn't tell me whether the parser was parsing the right thing. It could only tell me what "consistent" looked like.
It doesn't replace human judgment about bugs. The line-number case proved that. The model's invariant was reasonable and wrong for this context. I had to know the downstream system to make the right call.
And the free tier has constraints. The model's context window limited how much of the parser I could show it at once. I had to split the file into two prompts and merge the invariants manually.
Who should not use this: teams with existing test coverage and a clear spec. The invariants will be redundant. Who should use it: anyone facing a legacy module with zero tests and a refactor deadline. The model won't write your tests — but it will tell you what to test.
The lessons
Three things stuck with me.
First, asking for invariants instead of code changes the quality of the answer. When I ask a model to write a fix, I get plausible code. When I ask it to define correctness, I get a checklist I can verify. The second is more useful.
Second, a wrong invariant is still a gift. The line-number property was incorrect for this codebase, but it exposed a decision I needed to make. The test didn't tell me what to do — it told me there was a fork in the road.
Third, free infrastructure changes the workflow. I ran the suite hourly because I could. If I'd been paying per run, I would have run it once and moved on. The habit formed because the cost was zero.
If you've got a legacy file you're afraid to touch, don't ask a model to rewrite it. Ask it what "correct" means. Then encode that answer as tests. The refactor will still be scary — but at least you'll know when you've broken something.
Top comments (0)