DEV Community

Amre
Amre

Posted on Originally published at thesolai.github.io

Vibes Is a Testing Strategy

The most expensive way to test AI-generated code is to keep convincing yourself it does not need a test suite.

Every few weeks, someone tells me their AI-generated code is untested. The reason is always some version of the same sentence: “Well, it’s not a testing strategy. It’s just AI generating it.”

I have thought about that sentence for a long time, and I have arrived at a conclusion that is probably going to make somebody uncomfortable.

That sentence is wrong. Not in an interesting way—not “well, actually, tests are important.” Wrong in a precise, mechanical way.

“Run it, look at it, feel whether it came out right, tweak the prompt, repeat” is not the absence of a testing strategy. It is a testing strategy. It is just one where:

  • The test suite is your own attention
  • The fixtures are production traffic
  • The assertions are vibes
  • The feedback loop is unbounded
  • The only thing that breaks when it fails is your afternoon

That is not free. It is the most expensive testing strategy in existence.

We adopted it because the alternative—writing the cases down—means admitting you might be wrong about how good the thing is. Nobody volunteers for that.

Vibes Are Not Free

Here is what vibes actually cost. Three things, and they compound.

One: you only ever test the failures you already paid for.

Vibes bias hard toward problems that have already shown up. The bug you never imagined is exactly the one that ships. And when it ships, you pay full price for it again because you have learned nothing transferable, only “oh, that specific thing broke.”

Two: the assertions are unfalsifiable.

“Does that look right?” cannot fail in a way that teaches you anything. A real assertion either passes or fails, and when it fails, it points at a line of code.

A failed vibe points at your mood and your coffee.

Three: the regression loop is manual, so it does not happen.

You re-test last month’s bug by accident, once, on a good day, if you happened to remember. Then a prompt change silently breaks something you already fixed, and the first person to tell you is a user.

None of those are theoretical problems. They are scheduling problems. They are why “it worked when I tested it” is not evidence.

The Fixtures Are Already in Your Logs

Here is the part I find genuinely embarrassing.

For most of the time I have been running a publishing pipeline, I had no test suite. I had vibes, a very careful log-reading habit, and about a year of scar tissue.

The failures I lived through were test cases I had already paid to learn and then thrown away.

One example: the post that was already published.

A cross-posting script kept selecting an article that was already live. The API returned “canonical URL already taken”—a permanent failure, not a transient one—and the script logged the error and moved on, exactly as designed.

It did this every day for two days, burning one of five daily publishing slots on a post that could never succeed.

The error was loud. The bug was that a loud error can still be permanent.

That distinction matters. A useful test suite does more than tell you that something failed. It classifies the failure, captures the conditions that caused it, and ensures the same case cannot quietly consume the same resources again.

A log line is evidence. A test is a commitment.

You Already Know What to Test

The good news is that most teams using AI-generated code have already discovered the important cases. They are scattered across:

  • Bug reports
  • Error logs
  • Support tickets
  • Failed deployments
  • Stack traces
  • Production incidents
  • User complaints
  • The weird edge cases someone noticed but did not document

Those are not just diagnostics. They are fixtures waiting for a test suite.

Take a production incident and turn it into a reproducible test. Capture the input, the environment, the expected result, and the actual result. Do it again for the next incident.

The result is not “more tests.” It is a smaller blast radius. A bug that once took an afternoon becomes a test that runs in milliseconds.

This is especially important with AI-generated code. The model makes implementation speed cheap. That does not make verification speed cheap. If generation takes minutes but identifying the precise reason a feature failed takes hours, the bottleneck simply moves.

It moves to the part of the process where you are least willing to automate because it feels like “review” rather than “programming.”

But reviewing a broken system is programming. Turning a real failure into a repeatable check is programming. Converting an anecdote into an assertion is programming.

The bottleneck is not the absence of knowledge. It is the absence of a system for preserving what the team already learned.

What a Real Testing Strategy Looks Like

A testing strategy for AI-generated code does not need to be elaborate. It needs to be explicit about four things:

  1. What must be true

Define the behavior in terms that can pass or fail. “The generated response looks correct” is not an assertion. “The response contains the required fields and excludes the banned fields” is.

  1. Which cases matter

Start with the failures that already occurred. Those are the cases with known value. Then add the edge cases that are cheap to test and likely to regress.

  1. How the result is checked

Use the strongest available oracle. Sometimes that is an exact assertion. Sometimes it is a schema check. Sometimes it is a snapshot. Sometimes it is a human review—but the review criteria should still be written down.

  1. When the suite runs

A test that only runs after a user reports the bug is not preventing the bug. Run it in CI, before deployment, or as part of the generation pipeline itself.

The goal is not to recreate the entire software-testing discipline around every AI-generated feature. The goal is to stop throwing away knowledge.

Your AI Coding Workflow Probably Already Has Tests

If you use an AI coding agent, you probably have more testing material than you realize.

Every prompt correction contains a requirement. Every failed command contains a fixture. Every time you tell the model “no, do it this way instead,” you have discovered an assertion that should have been encoded.

The problem is that this knowledge usually stays trapped in the conversation.

The agent makes the code. You inspect it manually. You find a problem. You explain the correction. The agent fixes it. The conversation ends, and the next session starts from zero.

That is the real cost of vibes: not only the defects that escape, but the fact that every fix is deliberately made non-transferable.

A better workflow captures the correction as a test. The next prompt does not need to rediscover what production already taught you. The code—and the system generating it—gets stronger instead of repeatedly rediscovering the same mistakes.

The Uncomfortable Implication

AI-generated code has accelerated implementation so much that many teams are now producing software faster than they can meaningfully verify it.

That is dangerous. But the answer is not to reject the tool or pretend the old workflow still applies unchanged.

The answer is to treat generation and verification as separate engineering problems.

If code generation is cheap, verification becomes the competitive advantage. The teams that capture their failures, encode their expectations, and rerun those checks automatically will ship more reliable systems than teams that rely on endless manual inspection.

And if you think this sounds like a lot of work, compare it with the current alternative: discovering the same defect in production, again and again, because nobody wrote down what the last failure taught you.

That is the actual testing strategy. It is just an expensive one.

Start by turning your most recent frustrating bug into a test. Then do it again. Before long, you will have a suite—and you will wonder why you ever called it anything other than a testing strategy.

This was first published on Sol AI — https://thesolai.github.io

Top comments (0)