My test suite had been green for weeks. The product's main journey was broken.
TL;DR: a green test only proves what it actually checks. I lived through both worst cases. A green database test on a change that broke prod, because the test ran with full rights and prod did not. And a green UI test suite on a broken product, because it checked server responses instead of the screen. Now that AI writes 42% of code, verifying has become harder than writing. Here are the rules I apply since.
This article is for developers and leads who trust their green CI. CI is the machine that replays every test on each code change.
The setup
A personal project, but real-sized. Go services, a Postgres database, Kubernetes hosting.
The tests come in two families. Integration tests check a service against a real database. E2E tests, end to end, drive a real browser on the real product, the way a user would.
Part of the code is written with AI agents, then reviewed. Two incidents taught me the same lesson, at two different layers. A green test is necessary. It is not sufficient.
The green test that broke the deploy
A migration is a script that changes the structure of the database. It runs on its own, at deploy time.
A new service's migration carried one line too many: a command that changes the rights of a database account.
In production, the migration runs under an account with limited rights. That is on purpose, for security. So Postgres refused the command: that account has no right to change rights.
What follows is a cascade. The migration tool marks the database as "dirty". The service restarts in a loop without ever starting. Manual repair, with the administrator account.
The outage is not the worst part. The worst part is that the test was green. That test spins up a throwaway database in a container, with a tool called Testcontainers. And in that test database, the account is an administrator. Full rights.
ALTER ROLE app_system BYPASSRLS;
-- the command that changes an account's rights
-- test : superuser → OK
-- prod : restricted role → permission denied
-- schema_migrations.dirty = true → CrashLoopBackOff
So the forbidden command passed in test and broke in prod. I checked: the test was green on the broken version and on the fixed one. A test that passes on both sides of a bug tests nothing.
The rule since. The account the test uses has the same rights as the prod one. And a migration never touches accounts, only tables.
A green E2E suite on a broken product
Second incident, more embarrassing. For weeks, an entire journey was broken. A case created, a document signed, and the next screen showed nothing right.
The E2E suite? One hundred percent green, every day.
I audited the suite to understand. Almost half the tests never opened a browser. They called the API, the server's interface, and checked its raw response. Nobody was looking at the screen.
A test shortcut skipped the signing step. The very step those tests claimed to cover. And the tests of the real signing journey were marked "fix later". All of them.
Put differently: the front end could stop displaying the case state, the amounts, the next button. One hundred percent of the tests still passed.
Check what the user sees, not what the server returns
The suite's rewrite fits in one rule. Check what shows up on the screen. Never the server's raw response.
Why? Between the server and the screen sits the whole front end: the code that turns responses into pixels. That is what was broken. And no test was looking at it.
Two details that pay off. Checking every screen in both product languages catches missing texts, displayed as raw codes. And if you compare screenshots, always generate them on the same machine: text rendering changes from one machine to the next.
A shortcut on the tested path is a lie
The signing shortcut deserves its own section. It existed for a good reason: the real signature is slow and depends on an external service.
But a signing test that skips the signing does not test the signing. It tests that the shortcut works. It is a lie, neatly stored in the codebase.
And it hid something worse. In this product, the end of a signature fires an event. That event updates the states, sends the notifications, opens the discussion thread. The shortcut never fired that event. The state it built looked like the real one, without being it.
The rewrite rule: prepare data through the API for speed, fine. But the thing under test is never faked. The signing test signs for real, in the real module, all the way through.
2026: the hard part is no longer writing code, it is verifying it
Why talk about this now? Because the volume switched sides.
According to Sonar's State of Code survey, 42% of code committed today is AI-generated or assisted. 96% of developers say they do not fully trust it. And yet, fewer than one in two always checks that code before approving it.
Tests absorb the shock. New Relic reports that 78% of organizations see more production incidents since generated code arrived. CodeRabbit measures 1.7 times more issues in that code, and 75% more logic errors.
My two incidents tell the same story as those numbers. Code gets written fast now, tests included. A green suite has never been easier to produce. And has never proven less.
The checklist to test your test suite
Ask your green CI these questions. Each one comes from a real incident.
- [ ] The test's database account has prod rights, not full rights
- [ ] A test must fail on the broken version: verify that at least once
- [ ] Critical journeys are checked on the screen, not on the server response
- [ ] No shortcut on the path the test claims to cover
- [ ] API-prepared data fires the same events as the real journey
- [ ] Tests marked "fix later" are counted and visible, not forgotten
- [ ] Every screen is checked in every language the product ships
- [ ] AI-generated code goes through the same gates as yours, no exceptions
What to remember
A green CI is a precondition, not a proof. The useful question is not "do the tests pass?" but "what would break without them noticing?".
Audit your suite the way you would audit code: look for what it does not look at. In my case, the answer was: the product.
Want an honest audit of your test suite, or harder gates on generated code? Let's talk.
Sources: Sonar, State of Code Developer Survey (January 2026) · Sonar, The Current Reality of AI Coding · New Relic (June 2026) · CodeRabbit, State of AI vs Human Code Generation · PostgreSQL, ALTER ROLE
Top comments (0)