Behind every successful AI rewrite story, there is an unsung hero. And it's never the AI.
Chances are you've already come across the Reco story. Their AI tool rewrote JSONata from JavaScript to Go in a day or so. The end result? Half a million dollars per year less on infra costs. Insane. Blog gold.
Yet, I can't stop ruminating on the part no one really shared.
The real MVP wasn't the model
JSONata had an existing test suite. A good one. The type of test suite someone spent years working on, testing edge cases that'd likely give you nightmares.
That test suit is what made sure every line of Go code the AI squirted out was valid. Remove it, and it's not a $500k success. It's a 'hunch-based' reimplementation that probably works. Probably. 🤷
The AI didn't have a clue if the 'ported' code was correct. The tests did.
Speed without verification is just fast failure
The way it's presented is what bothers me. Saying "AI ported a language tool in days" makes it sound like the AI actually did all the engineering. But the porting itself isn't the complicated part. The complicated part is understanding whether the port was done correctly or not.
If you look at the process that way, you get:
→ AI generates thousands of lines of Go in hours
→ Engineers run the existing test suite against the output
→ Tests catch regressions, edge cases, type mismatches
→ Engineers fix what the AI got wrong
→ Tests pass, code ships
Remove second, three, and four, and you don't have anything left. You just have a nice autocomplete that, essentially, is creating a codebase that no one should have faith in.
We keep crediting the flashy tool
This situation is repeated in many places. A team uses AI to achieve something impressive, while the blog post starts with "We used AI". The actual testing infrastructure, the CI pipeline, all the regression tests that have been built up over the years are mentioned very superficially in the ninth paragraph.
It's like giving the bulldozer credit for the building at the construction site. Yeah, the bulldozer is fast, but the reason the building isn't just falling down is that somebody made blueprints and came in and inspected it.
I'm not bashing AI here. I enjoy and benefit from AI tools every day. They're great. But the model didn't magically save $500k because it's so smart. It saved $500k because someone, somewhere, likely years ago, wrote tests that caught the mistakes of an AI. That person will never have a viral blog post written about them. 😅
The uncomfortable takeaway
If your codebase doesn't have sufficient test coverage, AI rewrites are just rolling the dice. Not a strategy.
The teams getting real value out of AI assisted porting and migration are the teams that already did the hard work of testing. The AI just speeds up what good engineering was already making possible. It doesn't replace that.
→ No test suite = no way to validate AI output at scale
→ Strong test suite = AI becomes a genuine force multiplier
→ The investment in tests pays off in ways nobody predicted when they wrote them
The boring, unsexy work is always the load-bearing work. Always. 🏗️
So what now
So, next time you read an "AI saved us $X" headline, ask to see the test suite. Ask about their CI pipeline. Ask to speak to the engineer who spent a month of evenings in 2019 writing the edge case tests. That's where the real magic happens.
That's where the real story is.
What's the most underappreciated piece of engineering infrastructure on your team?
Top comments (0)