CI is so commonplace these days, we don’t often think about why it’s there. And that reasoning really matters these days, because of how we use it.
Especially when you’re writing AI agents and expect the CI system to run a regression test suite, and sound the alarm when something goes wrong.
Here’s a funny story you may not know. CI - Continuous Integration – was not originally a tool category. It’s a process - it’s how you continuously integrate pieces of code together.
Anyway, why do you need this process anyway? Because you’re scared. You’re terrified that integration broke something.
So what do you do? You look for approval. An automation system that runs all the tests all the time is our approval of choice.
That’s how CI became a tool category. Because in its heart, it’s a simple automation pipeline.
So, that automation’s holy grail is speed, right?
Wrong.
The mission was to kill “Works on my machine”
Remember when we wrote software in a cave, and we had a sticker on our computers – “Works on my machine. Don’t run it on another cave”?
Ah, the good old days.
Because the CI automation #1 OG mission was to kill “Works on my machine”. How? By creating a repeatable process. Independent of all the weird installs on my machine. Multiple versions of libraries. And different configurations. And admin hacks.
The CI builds, packages, runs and tests the software every time the same way. That way, if a test turns red, you know something bad happened, because it was green until now. And why your stomach turns, when it flakes between red and green – that’s the feeling of lost repeatability.
Yes, repeatability brings bliss. Well, it did, before AI.
Then AI showed up
If your app does not touch AI (although building with code agents, can sometimes count as “touch”), you’ll continue to feel that bliss.
But if you’re developing AI-based features, or agents, or vibe-coding, you’re in for a new experience every time you push your code (or prompts) into CI.
Things are not repeatable anymore. Models change, sometimes without warning. And always under your feet. And when a model changes, does it trigger a run?
You wish! Your alert system is malfunctioning.
And the worst part is not a regression. Although if it’s a bug in a prompt, you may not be able to fix it.
No, the worst part is that Green today doesn’t tell you what kind of Green it is. Things are no longer Working or not, they are Work-ish. But even that’s not the same work-ish every day.
So we’ve got fewer triggers, results we can’t read, and on top of it – every run means something else.
What’s the solution?
Repeatability gives us confidence. CI is the process, and tools, that give us the approval we seek, and the confidence we have comes from that repeatability.
We can’t rely on the trigger anymore, but we can initiate our own runs. And since one run is not enough, we need to run more and look at the trends. Catch drift before it ships.
What happens to quality when AI meets CI? We need to get back to the reasoning, and change how we manage quality.
Originally published at testingil.com.
I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.
Top comments (0)