Three failures I still think about

#programming #softwareengineering #career #debugging

People sometimes ask where the discipline comes from — the three-line rule, the habit of trusting nothing except my own running code, the assumption that production will surprise me. None of it is theory. Each piece came from a specific failure that reached real users and made me change how I work, permanently. Here are three I still think about. All three, I noticed later, are things I didn't do.

1. The collation I didn't know about

This was early. I was working on a content platform for Japanese feature phones — an official carrier service, the kind where charges land on your phone bill and the carrier has to approve you. User IDs were case-sensitive by spec. userA and usera were two different accounts.

The database was MySQL. The user lookup was a WHERE on a varchar column.

What I didn't know — what I didn't know was even a concept — was that the default collation made that comparison case-insensitive. WHERE id = 'userA' matched usera too. I had never set the column to a binary collation.

It ran fine for a long time. Then a user wrote in: points were missing from their account, and they had no memory of spending them.

I still remember the moment I understood it. Two accounts whose IDs differed only in case were colliding in the lookup. One person's spending was drawing down the other's balance. It had been happening quietly, to whoever shared a case-collision with someone active.

A few hundred users were affected before I found it. It remains the largest single mistake I have ever shipped.

The fix was one line of column configuration. The lesson was not about that line. The lesson was that I had built on top of a default I had never examined — and the default did not match the spec I was supposed to implement. After that, I stopped assuming I knew how a tool behaved. I check, especially the defaults, because the default is the part you never chose, and therefore never thought about.

2. The connection I forgot to release

A few years in. We were seeing access patterns that looked like abuse, and I wrote a process to catch them — flag suspicious source IPs, periodically push them into a blacklist. It was a small, defensive piece of code. I wrote it quickly and shipped it, because the abuse was happening now.

The logic was fine. Each time the process judged a request as suspicious, it opened a database connection, recorded the IP, and moved on.

It opened a connection. It did not always close one.

I had left the connection release out of the finally block. The pool didn't drain all at once — each flagged request leaked a single connection, so nothing broke at release time. The service ran normally while connections quietly disappeared in the background, one at a time. The more abuse we saw, the more connections we lost.

When it finally went down, the diagnosis was immediate. Something that had been working stopped working right after a release — there is only ever a short list of suspects, and I found the missing finally quickly. That part was not hard.

The painful part was the gap before it. Because the drain was slow, the outage didn't arrive until well after the release, by which point cause and symptom were separated in time. And I had believed I was watching the service. I wasn't — not in a way that would catch a gradual leak. The monitoring I thought I had was not the monitoring I actually had.

That is the part I still think about. The code I wrote to defend the service was the code that took it down — and the watch I thought I was keeping never saw it coming.

The fix was a finally block. The lessons were two. First: cleanup is not a step you add once the logic works — it is part of the code. Something that acquires a resource without guaranteeing its release in the same breath is not finished; it only looks finished, until the one path you didn't cover is the path that runs most. Second, quieter and slower to sink in: a gradual failure is more dangerous than a sudden one, because it separates cause from effect — and the monitoring you assume is protecting you may not be looking at the thing that matters.

3. The release note I didn't read

This one is recent — recent enough that I am slightly embarrassed it still caught me, this far in.

We relied on a third-party library for a core piece of what we produced. A new major version came out — a clean major-version bump, the kind you apply without much ceremony. I updated it, ran the unit tests, they passed, and I released.

Then a user wrote in: the output was coming out blank. Not an error, not a crash — a valid, empty result. The kind of failure that passes every automated check except the only one that counts, which is a person opening the file and seeing nothing there.

I had a guess immediately, and the release notes confirmed it within a minute. The new major version had changed an assumption the old one had quietly tolerated — about how the input and the output were allowed to overlap. My code had leaned on the old, lenient behavior for years. The unit tests passed because they exercised the shape of my code, not the behavior of the dependency against real data.

The fix was small. What changed permanently was how I treat that dependency — and every dependency like it.

I read release notes now, especially the section most people skip, the one labeled "breaking changes," because that is the version telling you in writing which of your assumptions it just invalidated. But reading is not trusting. A release note tells me what the maintainers chose to write down — not what they changed and didn't think worth a line. So I verify against the real service, every time. Not just on major versions: on minor bumps, on patch releases, on upgrades that have nothing to do with the part I care about. Especially the library that taught me this, which I now carry a permanent and slightly unfair suspicion of — the one I trust the least, and check the hardest.

It is an ungenerous way to treat a tool I depend on and need. But some libraries earn it, and the check costs me a few minutes while the alternative costs a user opening a blank file. So I run it and look at the real output myself, every time. Unit tests check the shape of my code. Only the real service tells me the truth.

What the three have in common

Three mistakes, spread across the years, with one shape in common: each carved a habit into me, and I have the habit because I have the wound. None of these is a principle I reasoned my way to. It is all scar tissue.

But there is one more thing, and it is the one that still makes me feel like a beginner after twenty-four years. I almost left it out, because it is the least flattering.

Every time something breaks — every single time — there is a flicker, less than a second long, where some part of me thinks: maybe this one isn't my fault. Maybe it's the library, the user, the infrastructure. And then, just as fast, I know better. It is almost always my fault, and I turn to the only useful question — what did I do? — but the flicker was there first. It is always there first.

Twenty-four years has not removed it. I know, completely, that when my code is involved and something is wrong, the suspect is me. And still the instinct to look away fires before anything else, and has to be overridden every time.

I have stopped expecting it to go away. It is probably just part of being the person who wrote the code — the small, animal reluctance to be the cause. What changed is not the instinct. It is the half-second. The override is faster now, that is all. Maybe that is what maturity is: not the absence of the flinch, but the speed of turning back toward the thing you would rather not look at.

That turn — toward your own code, toward the real output, toward the default you never checked — is where these three habits actually live.

Built with Claude (Opus).

Earlier in this series: