At 2:14 on a Tuesday afternoon, our signup error rate went from basically zero to 31%. It stayed there for nineteen minutes. Nothing was "down," exactly. Some requests worked, some returned a 500, and the dashboards looked like a barcode.
The cause was a migration I had reviewed and approved that morning. It renamed users.phone to users.phone_number. The new code read the new column, the tests passed, and the deploy went out as a rolling update. For nineteen minutes, half our pods were still running the old code, still querying a column that no longer existed.
Every test in that repo was green. Not one of them ran old code against the new schema, because not one of them touched a real database. I remember staring at the CI page during the incident call, scrolling through all those checkmarks, feeling oddly betrayed by a web page.
Integration testing, minus the textbook
Here's my working definition: an integration test checks that your code and something it doesn't fully control still agree with each other. That "something" might be a database, a broker, another team's service or a config file that only exists in production.
Unit tests check the logic inside a box. Integration tests check the seams between boxes. If you've never had to separate those two ideas in your head, a longer write-up on integration testing walks through the formal types and terminology. I'll keep this one to what I actually run and why.
The distinction matters because the failure modes are different. A unit test fails when your logic is wrong. An integration test fails when your assumptions are wrong, and assumptions are what change quietly while nobody is looking.
Why my green suite lied to me
That signup repo had over 600 unit tests. The repository layer was mocked in every one of them. So the tests didn't verify that phone_number existed in Postgres. They verified that our code called a function we had told them to expect. That's a closed loop: the code agreed with the mock because the same person wrote both.
This is my slightly unpopular opinion: a mock of your own database is usually a liability, not a test. It feels productive because the suite is fast and the coverage number climbs. But the mock encodes what you believe the database does, and your beliefs are the exact thing that needs checking.
It's getting worse, not better. AI assistants are very good at producing unit tests that agree with the code they just wrote. The 2024 DORA report estimated that as AI adoption went up, delivery stability went down by 7.2%. I don't think AI is the villain there. I think more code is shipping with tests that confirm intent instead of testing reality.
None of this means unit tests are bad. I still write them for parsing, pricing rules and anything with branching logic. I just stopped treating a green unit suite as evidence that a deploy is safe.
Where the bugs were hiding
Looking back at the incidents I've been paged for over the years, almost none were logic bugs. They clustered in four places.
Database migrations
The Tuesday story is the classic one. Renaming or dropping a column in the same release as the code change breaks any instance still running the previous version. The fix is boring: expand, migrate, contract. Add the new column, deploy code that writes both and reads the new one, backfill, and drop the old column in a later release.
What surprised me was how cheap it is to test. Run your migrations against a real database in CI, then run the previous release's integration tests against the migrated schema. If the old code breaks, your migration isn't safe for a rolling deploy. That single check would have saved my Tuesday.
API contracts between services
A billing service once added a new refund status, PARTIALLY_REFUNDED. Perfectly reasonable change. Our order service had a switch statement with a default branch that threw an error, written years earlier by someone who assumed the list was final. Partial refunds started crashing order history pages.
Both services had great test coverage. Neither had a test that exchanged a real payload with the other.
Message queues
Queues hide ordering bugs better than anything else I know. We had OrderCreated and OrderCancelled events landing on different partitions, so occasionally the cancel was consumed first. The consumer ignored a cancel for an unknown order, then happily created the order a few milliseconds later. Customers got charged for orders they had cancelled.
You won't find that with a mocked broker, because the mock delivers messages in the order you publish them. A real broker doesn't promise that.
Config and environment
Short one. CI ran in UTC. One production image had TZ set to a local zone by a base image we inherited. A "same day" check on refunds was off by one for a few hours every night. Integration tests that run on the same image you ship catch this. Tests that run on a developer laptop don't.
How I set it up now
My rule is: if my team owns it or deploys it, it runs for real in the test. If someone else owns it, I stub it at the network boundary.
That means Postgres, Redis and the message broker run in containers during the test run. A partner's payment API or an email provider gets an HTTP-level stub, and I refresh those stubs from real recorded responses every so often so they don't drift into fiction.
Here's the shape of a test I'd write today. It checks something a mock can't: that a unique, case-insensitive email constraint actually exists in the real schema after migrations run.
const { PostgreSqlContainer } = require('@testcontainers/postgresql');
const request = require('supertest');
const { buildServer } = require('../src/server');
const { runMigrations } = require('../src/db');
describe('POST /signup', () => {
let pg, server;
beforeAll(async () => {
pg = await new PostgreSqlContainer('postgres:16').start();
await runMigrations(pg.getConnectionUri());
server = await buildServer({ dbUrl: pg.getConnectionUri() });
}, 90_000);
afterAll(async () => {
await server.close();
await pg.stop();
});
it('rejects a duplicate email when only the casing differs', async () => {
await request(server).post('/signup').send({ email: 'dana@example.com' }).expect(201);
await request(server).post('/signup').send({ email: 'Dana@Example.com' }).expect(409);
});
});
A few things that keep this kind of suite usable:
- One container per test file, not per test. Startup is the expensive part. Each test creates its own rows with unique IDs instead of truncating tables, so tests can run in parallel without stepping on each other.
- Migrations run from zero on every CI run. If a migration only works on a database that already has last month's data, you want to know before production does.
- Integration tests run on every pull request. Not nightly. A nightly suite becomes a dashboard nobody opens.
- The CI image matches the production image. Same base, same timezone, same locale.
For service-to-service contracts, I don't spin up the whole fleet. That path leads to a 40-minute suite and a shared staging environment everyone is afraid to touch. Instead, the consumer keeps a small set of real payloads it depends on, checked into its repo, and the producer's CI replays them against its own API on every change. When billing adds a new refund status, billing's build goes red first, before the order service ever sees the payload. It's not glamorous, but it moves the argument to the pull request instead of the incident channel.
Then there's flakiness, which kills more integration suites than slowness does. Google's testing team reported back in 2016 that almost 16% of their tests showed some level of flakiness. If Google can't avoid it at their scale, you won't either, so plan for it.
My policy: a flaky test gets quarantined the same day, with an owner and a date. No silent retries. A retry policy turns a real race condition into a test that "sometimes needs a second run," and that's how the ordering bug above survived for a month on a team I worked with.
Mistakes I've made (so far)
Shipping a schema change and its code in one PR. Covered above, but it deserves the top spot because I did it more than once before it stuck.
Trusting a hand-written mock of a partner API for over a year. The partner had changed their error format twice in that time. Our mock still returned the old shape, and our retry logic never fired in production because it was looking for a field that no longer existed.
One giant seed file shared by every test. Somebody would tweak a row for their test and three unrelated tests would fail. We spent more time fixing fixtures than writing features. Per-test data, created inside the test, ended that.
Measuring the wrong thing. For a while we celebrated line coverage. It went up every sprint while incidents stayed flat. Once we started tracking "which incidents would an integration test have caught," the answer was most of them, and priorities shifted fast.
This isn't a new lesson for the industry, either. A NIST study back in 2002 estimated that software bugs cost the US economy about $59.5 billion a year, and that roughly $22 billion of that could be avoided with better testing infrastructure that caught defects earlier. The numbers are old. The pattern isn't.
The one habit worth stealing
If you change one thing this sprint, make it this: before your next schema change ships, run the previous version of your service against the new schema in CI. It takes an afternoon to set up, and it turns the scariest kind of deploy into a boring one.
I'm curious what's bitten you. What's the worst bug you've shipped that only existed between two systems, and what test would have caught it? Drop it in the comments. I collect these.
Top comments (0)