A few weeks ago, I mistakenly took my own site down for about thirty minutes.
I was wiring up a queue-based pipeline for GitHub webhooks. That needs two Cloudflare Workers: one that catches the webhook and drops the event on a queue, and a second that picks it up and does the slow work in the background. GitHub gives you ten seconds to respond, so you answer fast and do the real work after.
Two Workers means two deploys. I chained them into a single command.
Cloudflare's build system printed a warning. It said it was overriding the Worker name in my config with the name bound to the build project. One line, sitting in the middle of a wall of normal deploy output. Then the command exited zero, the way a successful command does.
What that warning actually meant is that my second Worker, all 0.39 KiB of it, had just been uploaded as a new version of the first one and promoted to live traffic. That Worker knew how to read from a queue. It had no idea what an HTTP request was. Every request to the site threw an exception until I worked out what had happened.
Thirty minutes of downtime. About two hours all in, from the bad deploy to a rebuilt setup that can't do that to me again.
Nothing about it looked wrong. The config was fine. The command was fine. The output scrolled by looking like every other deploy I've run. The one line that could have saved me was formatted like the ones around it. It didn't stop anything, because it wasn't an error. It was information.
The risk was there the whole time. Nothing showed it. Then traffic hit it.
That's not how it works in the physical world.
Walk past a construction site. You can see the risk. Someone on the fourth floor with nothing behind him. A crane swinging a load over the sidewalk. A beam carrying more than it looks like it should. You don't need a process or a meeting to notice any of that. The risk is right in front of you. Everyone who walks past does some version of assessing it, not just the person who signed off on the plans.
Software has none of that. There's nothing to look at. A risky change and a safe change are the same thing on your screen. They're just text.
So the risk stays in someone's head. Never on paper, never in the ticket.
And you know the feeling. Someone's describing a ticket in planning and something in your gut says this one is going to be worse than it sounds. Maybe you say so. Maybe it turns into an extra point on the estimate. It almost never gets written down anywhere as a fact about the work, somewhere you can look at later.
That invisibility does something worse than hide risk after you ship, though. It changes what you build in the first place.
Put a team under a real deadline and watch what gets cut. It's never the feature. The feature is the visible thing, the thing that goes in the demo. What gets cut is the test for the weird path, the handling for when the third-party call times out, the ten minutes of thought about what happens when this runs at ten times the volume. That work just vanishes, because nobody sees it not happening. Nobody demos the edge case they handled.
And "we'll fix it next sprint" only ever covers the misses you can see. The dangerous ones pass the quality bar looking fine. They ship green. Then they surface on their own schedule, weeks or months later, as an incident.
Which means the corner you cut and the incident it caused almost never land in the same sprint. From the inside they look like two unrelated events.
So what do you actually do about it?
Make it visible. That's the whole job.
Not a new process or a committee. Just moving it out of your head and onto the work itself, at the level of the individual change. This change touches auth. That one has no test for the failure path. Another lands in a part of the system that broke twice this quarter. Write it down where someone else can see it.
Once it's visible, two things change. You can weigh it, which is what the construction site gets for free. And you can defend it when the deadline shows up. The work on the chopping block finally has a name. Instead of being a vague feeling that loses every argument against a feature you can demo.
That warning was in my deploy log the entire time. It just looked like everything else.
Top comments (1)
"The one line that could have saved me was formatted like the ones around it" — that's the whole incident in one sentence, and the formatting is only half of it. The other half is the exit code. A warning that doesn't stop the deploy is a message that requires a human to be reading carefully at exactly the wrong moment, and nobody reads carefully during a routine deploy. That's not a discipline failure; the process is designed around not needing to.
I deploy to the same platform and I've paid for the reading-required assumption in a slower way. I have a scanner that checks my own code and prints a report. It was green for weeks while silently mis-scoring six clean files, and I never opened it — not because I don't check, but because green never asks to be opened. My attention only fires on things that visibly demand it, which is exactly the class your warning wasn't in.
What changed things for me wasn't making warnings louder, it was converting the ones that matter into non-zero exits. If a check finds something I'd want to know about, it fails the run rather than mentioning it. That reframes the question from "did I notice" to "did it let me continue," and only the second one is answerable after the fact. Your case is the strongest possible argument for it: exit code zero on a deploy that had already replaced one Worker with another. The information existed; the only thing missing was refusal.