It Had Been Working Fine

Four people apologized. None of them was lying.

The Screenshot

It started with a screenshot.

A user was testing the product. They called an endpoint, and the response came back wrong — an error about an ID that should not have been missing. Then the endpoint stopped working altogether.

We checked the logs. The table behind that API had no AUTO_INCREMENT on its primary key. Every insert was failing.

The release was pulled. The fix would take one line.

Four Apologies

The developer. "I'm sorry. I missed an AUTO_INCREMENT on the table. I'll add it."

The reviewer. "I'm sorry. I didn't read it carefully. It was a small oversight."

The tester. "I'm sorry. I haven't really been doing much lately. When I push, people find me annoying. So I stopped pushing."

The project manager. "I'm sorry. We don't have enough people. I know the process isn't being followed, but I can't hire."

Four apologies. All true. None of them an explanation.

The Second Round

The first time, they told us what happened. The second time, they told us why.

The developer, pressed. "In daily work we only run SELECTs. Simple queries. This time I needed a new table, so I exported the definition with SHOW CREATE TABLE, ran it in the database, and it worked. I checked it many times. I never found the problem."

The reviewer, pressed. "I'm very busy. Developers ping me to review and merge. I agree, and I merge. I don't have time to read every line."

The tester, pressed. "I'm very busy. I can't watch every team's testing. Some test tasks get pushed down to the developers themselves. So in practice, the developers don't test either."

The project manager, pressed. "We ship on a schedule. Everyone knows the process. Nobody has time to follow it."

The Quiet Months

The system had no API tests. No integration test between the application and the database. No real review of SQL. And it had been running for months. It worked fine.

Why? Because the work was simple. Only SELECTs. Simple systems are forgiving. You can run one for a long time with no tests, and it will quietly teach you that you have a process.

You don't. You have simplicity.

Then the first non-simple change arrives — a CREATE TABLE — and simplicity stops covering for you.

The Point of Bureaucracy

We complain that bureaucracy is slow. We rarely ask why.

A sound system is slow on purpose. Before anything can move, it has to pass through several reviews, several signatures, several people who are allowed to say no. Each of those steps is an interception point — a place where one person's mistake can still be stopped before it becomes everyone's problem.

Code works the same way. Tests and reviews do not exist to help you ship faster. They exist so that when something goes wrong — and something always goes wrong — it goes wrong small. Security people have a name for this: the blast radius. A good process is a wall built to shrink it.

A good system is not designed to make good things happen quickly. It is designed to make bad things matter less. Its job is not to speed up the best outcome. It is to contain the worst one.

We had neither. No slowness, no wall. One missing line, one unreviewed statement, and an entire endpoint died — not because the developer was careless, but because nothing stood between his mistake and production.

That is what the slowness was for.

What to Do

The lesson is not "add more process." Process without enforcement is theater, and theater is exactly what we had.

Test the system, not the SQL. The developer confirmed the query ran. Nobody confirmed the feature worked. Those are different questions, and only one of them matters.

A review that isn't read is worse than no review. It manufactures confidence. If reviewers are too busy, the answer is capacity, not a reminder to be more careful.

Ask what your simplicity is hiding. "It works" is not evidence that it is safe. It may only be evidence that nothing has challenged it yet.

Epilogue

The four people we interviewed were not villains. They were tired, understaffed, and doing exactly what the system rewarded: checking boxes.

Everyone did their job, in the sense that the boxes were checked. Nobody did their job, in the sense that mattered.

The bug was one missing line.

The cause was a system that had learned to run without anyone looking.

It had been working fine. That was the problem.