This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
I build a SaaS for small businesses. Restaurants, small shops, the kind of owner who needs to track sales, stock, cash flow and finances without wrestling a heavyweight ERP. For the first months I was heads-down on features, because that is what pays: the product has to solve a real problem the day you ship it.
Then one night taught me something no feature ever could.
The mistake was entirely mine
During a deploy, a configuration of my own doing exposed a payment API key in production. The exposure window was short, but the risk was not theoretical. Anyone who found that credential could try to use it. In production. Against real money.
Here is the part I got right, and it was as much luck as foresight: weeks earlier I had built an emergency mechanism for exactly the kind of unlikely event I did not believe would ever happen to me.
Every payment integration ran behind a monitor that watched for abnormal usage. A sudden jump in transaction volume. Calls from origins it did not recognize. Sharp changes in behavior. Anything outside the profile it expected. If any of those tripped, the integration did not wait for a human. It moved itself into a protection mode, blocked new operations, and paged me immediately.
That is exactly what happened.
The circuit breaker did its job
The monitor flagged the anomaly and locked the integration into protection mode before a single improper transaction went through. The alert woke me in the middle of the night. I revoked the compromised credential, generated a new key, rolled it across the whole infrastructure, and validated everything before the first customer opened the app in the morning.
No customer was affected. No improper transaction went through. No financial data was exposed.
And none of that made the mistake any smaller.
What I actually learned
The lesson was not "be more careful." Careful is not a strategy. I had been careful, and I still shipped the mistake at the worst possible layer.
The real lesson was that security cannot rest on not making mistakes. It has to assume mistakes will happen, and it has to cap the blast radius when they do. The thing that saved me that night was not my discipline. It was a layer I had built specifically to survive my own discipline failing.
So I rebuilt the security architecture around that idea:
- Automated rotation of sensitive credentials, so a leaked key has a short life by default.
- Hard segregation between development and production, so the two can never bleed into each other.
- Detailed audit logging of critical events, so every sensitive action leaves a trail.
- Extra validation during deploy to stop secrets from being exposed in the first place.
- Continuous monitoring of every external integration.
- Real-time alerts for anything that looks abnormal.
- A mandatory security review before any release reaches production.
The question I ask before I ship anything now
Every new feature I build now starts from one question:
If I make a mistake tomorrow at 3 a.m., will the system protect my customers before they notice?
That night changed what engineering means to me. Building is no longer just shipping features. It is building systems that expect human failure and protect the people who trust them anyway.
The bug that night was mine. The thing that smashed it was a layer I had built for a version of me that would eventually slip. Build that layer before you need it. You will need it.
Top comments (2)
The blast-radius framing is the right instinct, but I'd put more weight on the key's scope at issue time than on the breaker at abuse time. A circuit breaker trips on anomaly, and the nastier leaked-key abuse is low-and-slow — a handful of charges an hour that sit under your baseline until the monthly total surprises you. Short TTLs and a narrowly-scoped key cap the damage deterministically, where anomaly detection just makes a patient attacker pace themselves under the threshold. Detection's your second line here, not your first.
You're right, detection is the second line, not the first. The breaker caught that night because the abuse was fast and loud. Low-and-slow walks right past it.
The one control that doesn't care about pace is reconciliation, not anomaly. Every charge should map to an intent I recorded before the call. A charge with no matching intent is fraud at one an hour or a thousand a minute. Rate detection asks "is this faster than normal." Reconciliation asks "did I authorize this at all," and low-and-slow has no answer to the second question.
So the order I'd write now: short-lived scoped keys first, reconciliation second, the breaker third. It earned its place in the story, just not first.