There's a particular kind of meeting that happens in every engineering organization eventually. Someone puts a slide on the screen showing quarterly infrastructure spend. The numbers are climbing. A VP — almost always someone whose mental model of software was formed before Kubernetes existed — asks why the monitoring bill is larger than the compute bill. The room goes quiet in a specific way. The engineers know the answer. They're trying to figure out whether this is a safe room to say it in.
That silence is where reliability goes to die.
I've been in enough of those rooms to have developed a kind of diagnostic reflex. When I hear the phrase "right-size our observability footprint," I mentally note it the way a cardiologist notes a patient describing occasional chest tightness. Could be nothing. Probably isn't nothing. The framing itself — observability as a footprint to be right-sized — reveals a category error that will eventually cost ten times whatever the proposed savings are. But I've also learned that explaining this at the moment of the meeting rarely works. The incentive structures around that table are arranged against you, and they've been arranged that way deliberately, often by people who would insist they care deeply about uptime.
Reliability is not primarily a technical problem. It never was. The systems I've watched fail catastrophically over the years — the multi-hour outage that eviscerated a Black Friday revenue quarter, the cascading database failover that turned a 15-minute incident into a 6-hour one because nobody had actually tested the failover path since 2019, the gradual degradation nobody noticed because the dashboards had been thinned to cut Datadog costs — almost none of them failed because the engineers didn't know how to build resilient systems. They failed because the organization had been making a long series of individually defensible decisions that collectively eroded the architecture's tolerance for surprise.
This is worth sitting with. Individually defensible. That's the insidious part.
What Resilience Actually Costs, and Why the Bill Is Always Surprising
Let me be concrete about what investing in reliability requires, because the abstract framing of "multi-region infrastructure" and "redundant pipelines" obscures what you're actually buying.
Multi-region deployment means you are running your entire production topology — compute, storage, networking, secrets management, deployment tooling — in at least two geographically separated environments simultaneously. You are paying for idle capacity that exists specifically to absorb a failure you hope never occurs. You are maintaining parity between those environments as your application evolves, which means every deployment, every schema migration, every config change has to be tested against a more complex matrix of states. You are dealing with the genuinely hard distributed systems problem of data consistency across regions, which introduces latency trade-offs and replication lag that your application code must be written to tolerate. And you're doing all of this while your product team is asking why the new checkout flow hasn't shipped yet.
That's before you touch the human infrastructure. Chaos engineering — real chaos engineering, not the theater version where you run a tool against a staging environment and check the box — requires dedicated engineering time to design failure scenarios, execute them against production, analyze the results, and iterate. It requires an organizational culture where engineers aren't penalized for the incidents those experiments occasionally produce. It requires incident rehearsal, which means pulling senior engineers out of feature work to run tabletop exercises for failure modes that may never materialize. The return on that investment is invisible until the moment it isn't, and by then, you're too busy fighting the incident to feel grateful for the preparation.
The bill compounds. That's the thing organizations keep rediscovering. It's not a one-time investment. Resilience requires maintenance the same way a physical plant requires maintenance. The runbooks go stale. The on-call rotation gets thin when engineers leave and aren't replaced at the same rate. The canary deployment that was carefully tuned to catch regressions silently starts evaluating against the wrong metric after a service rename. I've inherited systems where the "comprehensive observability" was technically still in place — the dashboards existed, the alerts were configured — but nobody had reviewed them in 18 months and half the panels were displaying data from a service that had been deprecated.
For early-stage systems, none of this makes sense. The blast radius of a startup's outage is bounded by the number of users who are currently trying to use it, which is probably small, and the cost of the outage is mostly reputational in a space where users have modest expectations. The right call at that stage genuinely is to move fast. The mistake is failing to notice when that calculus changes.
Reliability Debt: How It Accumulates, Why It's Invisible Until It's Catastrophic
Technical debt gets discussed constantly. Reliability debt is the same phenomenon operating in a different register, and it's harder to see because the damage doesn't manifest incrementally — it manifests in sudden, expensive discontinuities.
Here's the mechanism. Over the course of a year, your team makes a dozen small decisions under resource pressure:
You consolidate three microservices onto a single compute cluster to reduce costs. This makes sense — those services were underutilized, the consolidation is clean, and you save a meaningful amount on cloud spend. What you've also done is increase the blast radius of any incident affecting that cluster. Previously, a bad deployment to service A didn't threaten services B and C. Now it might.
You reduce the sampling rate on your distributed traces from 100% to 10% to bring the observability bill down. Still enough data to analyze normal operation. But under failure conditions, when you need the traces most, you're now looking at a 10x coarser picture of what your system is doing, and the specific transactions that are failing may fall into the 90% you're not capturing.
You remove the staging environment because it was perpetually out-of-date with production anyway, and the engineers weren't using it consistently. Fair criticism of the staging environment as it existed. But now the first place your changes encounter production-like traffic is production.
You centralize IAM permissions to simplify the access management overhead. Fewer roles, cleaner policy documents, easier auditing. Also, when a credential is compromised or a permissions bug ships, the scope of what that principal can affect has grown.
None of these decisions is obviously wrong in isolation. Some of them have genuine merit. Collectively, they have reshaped your system's failure profile in ways that are almost impossible to hold in your head simultaneously. The architecture is now operating with higher blast radii, lower diagnostic resolution, reduced isolation between deployment paths, and broader credential scopes than it was a year ago. It looks mostly the same. It is substantially more fragile.
The debt shows up in a specific way: the system's first serious incident will be worse than it should be. The response team will lack the observability data they need to diagnose quickly. The failure will spread further than it would have if the blast radii hadn't expanded. The recovery will take longer because the staged rollback path has degraded. And by the time the post-mortem happens, it will be genuinely difficult to attribute the severity to any individual decision, because each decision was reasonable and the compounding effect was never explicitly modeled.
This is why I'm skeptical of cost optimization work that doesn't explicitly surface reliability implications. Not because the optimization is wrong, but because the reliability implications are real and need to be priced into the decision. The question isn't "does this save money" — it's "does this save money after accounting for the change in expected incident cost."
The Incentive Structure Problem Is Not Fixable Through Awareness Alone
I want to be careful here, because the standard move in reliability discourse is to identify the incentive misalignment and then imply that organizations simply need to do better at recognizing it. As if the problem is primarily cognitive. As if the VP who asked about the observability bill just needs to be educated.
That framing is comforting and largely wrong.
The incentive structures that deprioritize reliability are doing exactly what incentive structures are designed to do. Engineering teams are measured on delivery velocity, feature throughput, deployment frequency. Reliability investments don't move those metrics. In fact, they temporarily depress them — time spent on resilience work is time not spent shipping features, and every hour of chaos engineering that doesn't produce an incident looks, in retrospect, like a particularly expensive way to confirm the system was working fine. The engineers who are most disciplined about reliability investment are the ones whose work is hardest to justify in planning cycles, because the counterfactual — what would have happened if they hadn't done it — is invisible by construction.
The people making prioritization decisions are not, for the most part, irrational. They're responding to the measurement system they operate in. You can run all the awareness campaigns you want about reliability debt; until the measurement system changes, the behavior won't.
What actually works, in my experience, is translation. Not education about reliability engineering concepts, but translation of risk into the language the organization already speaks.
When I've had success getting reliability investment funded, it's been by doing something unglamorous: modeling the expected cost of specific failure scenarios. Not in abstract terms — not "a multi-hour outage could be costly" — but as concretely as the data allows. How many transactions per hour does the checkout flow process? What's the average transaction value? What's the support ticket cost per affected user? What does engineering time cost during an extended incident? What's the realistic time-to-detection under current observability coverage, and how does that change under reduced coverage? What's the mean time to recovery if the on-call engineer can see 10% of traces versus 100%?
These numbers are imprecise. But imprecise numbers are far more persuasive than precise abstractions, because they put the discussion on the same ground as the budget conversation. You're not asking for reliability investment as a matter of engineering principle. You're presenting a comparison of expected values. The proposed cost reduction saves X per month. The change in risk profile costs an estimated Y per month in increased expected incident severity. The sign of X minus Y determines whether the optimization makes sense.
Sometimes the math favors the optimization. Sometimes it doesn't. Either way, you've moved from a values argument to a financial one, and financial arguments are winnable.
The Post-Incident Investment Cycle: A Pattern Worth Naming
There's a cycle I've watched repeat so many times that I've started thinking of it as a law rather than a tendency.
Phase one: the system is operating with insufficient resilience. The team knows this, but the investment to fix it keeps getting deprioritized. The system doesn't fail obviously, which makes it easy to argue that the risk is acceptable.
Phase two: a significant incident occurs. The kind that generates a post-mortem with an executive summary. The kind that results in a retrospective where someone says "how did we not catch this earlier" and everyone in the room understands, but nobody says out loud, that the answer is "because we kept deferring the work that would have caught it."
Phase three: emergency investment. Suddenly there's budget. Headcount gets pulled from feature work. Platform improvements that have been in the backlog for 18 months get prioritized. New observability tooling is evaluated and purchased. An incident commander role gets created. The SLOs that existed on paper get actual alerting attached to them.
Phase four: stability. The system is genuinely more resilient now. The incident rate drops. Post-mortems start looking less severe. The engineering organization feels like it's operating with more slack.
Phase five: the pressure shifts back. The business environment hasn't changed. Investors or board members are still asking about delivery velocity. Product roadmaps are still packed. The phase three investments improved reliability, but they also increased operational cost and consumed engineering capacity. Gradually — through attrition, through deprioritization, through legitimate competing pressures — the reliability investments begin to erode.
And then you're back in phase one.
I don't think this cycle is inevitable, but I think breaking it requires organizational maturity that's genuinely rare. It requires leadership that holds reliability investment constant through the stability phase, not just through the crisis phase. It requires measurement systems that make reliability degradation visible in real-time, not just after a major incident. It requires engineers who are willing to sound alarms about accumulating debt before those alarms are obviously warranted, which means being willing to be wrong sometimes and dealing with the credibility cost of that.
The organizations that escape the cycle are the ones that have internalized a particular idea: stability is a product, not a state. It requires ongoing investment to maintain, just like any other product. The moment you stop investing in it, it begins to decay.
Observability Is Infrastructure, Not Monitoring
I want to dwell on the observability economics question specifically, because it's where I see the most consequential misunderstanding in practice.
The framing of "observability as a monitoring cost" treats telemetry as a reporting layer that sits on top of the real system. Under this framing, you can tune the telemetry fidelity to manage costs without affecting the system's behavior. The metrics and traces and logs are just recordings of what happened; the system itself is unchanged.
This is technically true and operationally misleading.
What observability actually determines is your ability to operate the system. Not to run it — to operate it. To detect when it's behaving unexpectedly. To diagnose the root cause of a failure. To verify that a change produced the intended effect. To distinguish between correlated failures and coincidental concurrent failures. To understand whether a gradual degradation represents a trend or a transient fluctuation.
When you reduce telemetry fidelity, you don't reduce the frequency of incidents. You increase the time it takes to detect and diagnose them. And time is money in ways that are more concrete than the telemetry cost itself.
The numbers I've seen in practice suggest that a 2x increase in mean time to detection, compounded by a 1.5x increase in mean time to diagnosis, produces an incident cost multiplier that substantially exceeds the telemetry savings. But these numbers are organization-specific, and you have to actually compute them. Asserting the principle in a budget meeting doesn't work. Showing the calculation does.
There's a subtler point underneath this. Observability isn't just useful during incidents. It's what allows engineers to move quickly with confidence during normal operation. The ability to deploy a change and immediately see whether it affected error rates, latency distributions, and business metrics is what makes fast iteration safe. Organizations that cut observability in the name of enabling velocity often produce the opposite effect: engineers move more slowly because they're less confident, or they move quickly and break things that wouldn't have broken if they'd had better feedback loops.
Visibility is expensive. Blindness is more expensive. This isn't a slogan. It's a calculation, and I've watched enough incidents to have substantial confidence in the direction of the inequality.
The Leadership Translation Problem
Senior engineers and engineering leaders occupy an uncomfortable position in reliability economics. They understand the technical risk clearly enough to know what should be done. They often lack the organizational authority to simply do it, which means they need to advocate for it effectively.
Most technical advocacy fails not because the substance is wrong but because it operates in the wrong register. When an engineering leader says "we need multi-region redundancy," the implicit model is that leadership will evaluate the technical argument and reach the right conclusion. But leadership is already operating with a full attention budget, a financial planning cycle with constraints, and a set of competing priorities that are also being framed compellingly. The reliability argument is one of many, and "we need redundancy" competes poorly against "we need to ship this feature to capture this market."
The translation requirement is genuine. Technical risk has to become business exposure. "A single-region outage could halt revenue generation" is better than "we need redundancy," but it's still abstract. Better is: "We processed 200,000 transactions last quarter in our primary region, with an average transaction value of $85. Our historical MTTD in single-region failure scenarios is approximately 40 minutes, and MTTR has averaged 2.5 hours. That's a potential $42 million exposure per incident that our current architecture isn't isolated from." Now you're in the same room as the CFO's model.
This translation work is uncomfortable for many engineers, for good reasons. It requires making estimates with uncertain inputs and presenting them with false precision. It requires framing technical concerns in financial terms that feel reductive. It requires operating as an advocate rather than as an analyst. None of these are natural modes for people who were attracted to engineering because of its rigor and objectivity.
But I've watched the alternative play out. Technical correctness without business fluency results in reliability work being perpetually deferred, periodically rediscovered after major incidents, and never institutionalized. The pattern repeats indefinitely.
What Monday Morning Looks Like
If you've read this far and found it resonant rather than abstract, you're probably sitting with a system that has some of what I've described: accumulated reliability debt, under-instrumented failure paths, observability that's been thinned for cost reasons, a post-incident investment backlog that hasn't been touched since the last crisis, incentive structures that don't reward the work you know needs doing.
What do you actually do?
First: make the debt legible. Not to advocate for fixing it — not yet — but to understand it. Take the time to write down, concretely, what your current architecture's blast radii are. Which component failures are isolated? Which ones are shared? What's the observability coverage on your critical paths? When did you last actually test your failover procedures? Not document them. Test them. Meaning run them against a production-like environment and time the recovery.
Second: identify the three failure scenarios that would hurt the most. Not the most likely — the most consequential. Price them. What does an hour of that specific failure actually cost, in revenue impact, engineering time, customer support volume, and reputational exposure? Keep the numbers rough. Rough-but-honest beats precise-but-fake.
Third: find the cheapest reliability investment you haven't made yet that would materially reduce one of those three scenarios. Not the comprehensive solution. The cheapest meaningful step. This is where the argument for action becomes easy to win, because you're not asking for the full investment — you're asking for the part of the investment that has the best return.
The full program of resilience takes years to build and requires sustained organizational commitment that has to be re-earned continuously. But there's almost always something you can do this week that moves the needle on your most acute risk. And doing that thing builds the track record of reliability work delivering value, which is the prerequisite for getting the bigger investments funded.
Every architecture reflects what an organization actually valued, not what it said it valued. The post-mortem document where everyone agrees that observability is critical doesn't mean much if the next budget cycle cuts the telemetry spend. The reliability roadmap that's been in the planning doc for two years without a single item getting built is telling you something accurate about organizational priorities, regardless of what people say when the topic comes up.
Outages are revealing in this way. They expose not just what failed technically, but what the organization had implicitly decided it was willing to risk. The failure was authorized, in a sense — not deliberately, not in any single decision, but through the accumulated weight of choices about where to spend time and money and attention.
The organizations that build durable reliability aren't the ones that respond best to incidents. They're the ones that have made the reliability investments boring because they're consistently funded, consistently practiced, and consistently measured — even when there's no visible crisis demanding them.
That's harder than it sounds. It requires leadership that can hold a long time horizon through short-term pressure. It requires engineers who can translate risk into business language without losing technical precision. It requires incentive structures that reward prevention rather than just heroic recovery.
But it's achievable. I've seen it. The organizations that get there tend to share one characteristic: they treat reliability as an obligation to the people who depend on their systems, not as a cost center to be optimized.
That framing change — subtle, almost philosophical — turns out to matter enormously when the quarterly infrastructure review comes around and someone puts the observability bill on the screen.
Top comments (0)