The alert fired at 2:47 AM. Memory usage on the primary database cluster hit 92%. The on-call engineer saw it. The SRE lead saw it. The platform team had dashboards showing exactly which queries were consuming resources, how long the spike had been building, and what the projected failure point would be.
By 3:15 AM, no one had restarted anything, killed any processes, or scaled the cluster.
Not because they didn't know what to do. Because no one was certain they had authority to do it without checking with someone else first.
The on-call engineer could restart services but wasn't authorized to scale infrastructure without approval. The SRE lead could approve scaling but wanted to confirm it wouldn't blow the monthly budget. The platform team could provision resources but needed sign-off from the VP of Engineering for anything that affected production during business hours in APAC.
By the time everyone aligned, the issue had resolved itself. The batch job finished. Memory dropped back to normal. The postmortem noted "alert response time could be improved." But the real issue wasn't speed—it was that no one knew who owned the decision when the data said act now.
Information availability isn't decision authority
We've observed this pattern across dozens of engineering orgs. Teams invest heavily in observability, monitoring, and analytics. They build dashboards that update in real time. They configure alerts with sensible thresholds. They deploy machine learning models that predict capacity needs, detect anomalies, and flag performance regressions.
And then decisions still take hours, sometimes days.
The assumption is usually that better data leads to faster decisions. If engineers can see what’s happening, they’ll know what to do. But that’s only half the equation. The other half is who’s authorized to act on what they see. This gap is at the heart of operational decision support systems may surface the right information, but without clear ownership, decisions still stall.
In most organizations, that authority is fuzzier than the architecture diagrams suggest. Someone might be on-call, but only for certain services. Another person can approve infrastructure changes, but only under specific conditions. A third person has budget authority, but doesn't get paged for incidents.
When an alert fires, the data is clear. The decision path is not.
The dashboard everyone watches but no one acts on
We've seen this play out in capacity planning. A platform team builds a forecasting model that predicts when they'll need to add nodes to the Kubernetes cluster. The model tracks resource usage trends, seasonal patterns, and growth trajectories. It flags when capacity will hit 80% in the next 30 days.
The team sees the warning. They agree the forecast looks accurate. Then nothing happens for two weeks.
Why? Because provisioning new infrastructure requires a purchase order. The PO needs finance approval. Finance wants to see a cost-benefit analysis. The cost-benefit analysis needs input from product on expected user growth. Product needs to confirm with sales. Sales is waiting on Q4 pipeline data.
The model was right. The organization just wasn't structured to act on it within the window where action would have been useful.
Eventually, capacity hits 85% and becomes urgent. Someone shortcuts the approval chain. The nodes get provisioned. And the team adds "improve capacity planning" to their quarterly goals, even though the planning was fine—the decision process wasn't.
When incidents become committee decisions
Incident response should be fast. The whole point of on-call rotations, runbooks, and service-level objectives is to enable quick action when things break. But we've watched even well-instrumented incidents slow to a crawl when decision ownership isn't clear.
A microservices architecture starts throwing timeout errors. The logs point to a specific service. The metrics show it's overloaded. The on-call engineer has three options: restart the service, scale it horizontally, or fail over to a backup region.
In a well-defined system, the engineer picks the appropriate response and executes. But in many organizations, each option requires different approval. Restarting might be fine. Scaling costs money and needs infrastructure approval. Failing over affects multiple teams and requires coordination.
So the engineer opens a Slack thread. Tags the relevant people. Explains the situation. Waits for consensus. By the time everyone agrees on the approach, the service has either recovered on its own or the incident has escalated to the point where someone senior just makes the call.
The data was available the whole time. The metrics, logs, and traces all pointed to the problem and the solution. What was missing was clarity on who could decide to act.
Release decisions that stall despite green builds
We've observed similar patterns in release management. A team runs automated tests. The build passes. Code coverage looks good. Performance benchmarks are within acceptable ranges. The feature has been reviewed and approved.
And then the deploy sits in a queue for three days.
Not because anyone thinks it's risky. Because the release process requires sign-off from QA, product, and customer success before pushing to production. QA is waiting for product to confirm the feature is still a priority. Product is waiting for customer success to verify there are no open escalations that might be affected. Customer success is waiting for the account team to confirm the timing won't disrupt a major customer demo.
The system said "ready to deploy." The organization said "wait for alignment."
This isn't a technical problem. The CI/CD pipeline works fine. The issue is that the pipeline can't encode organizational dependencies. It can tell you the code is safe to deploy, but it can't tell you whether all the stakeholders who need to weigh in have actually weighed in.
So releases slow down, not because the data is unclear, but because the decision authority is distributed across people who aren't in the deployment flow.
Customer risk signals that no one owns
A more subtle version of this happens with customer health scores and churn prediction. An ML model flags an account as high-risk. Usage has dropped 40% over the past two weeks. Support tickets are up. The customer hasn't responded to the last three check-in emails.
The data lands in a dashboard. The customer success team sees it.
They agree the account is at risk. But who should reach out? The CSM doesn't have authority to offer discounts or expedite feature requests. The account executive could, but they're focused on renewals and new deals. Product can't make promises about roadmap priorities without engineering buy-in.
So the account sits in the "at-risk" column. It gets discussed in weekly meetings. Someone usually says "we should do something about that." And then the customer churns.
The model did its job. The prediction was accurate. The organization just didn't have a clear owner for acting on customer risk signals that required cross-functional coordination.
Why alerts become discussion triggers instead of action triggers
Over time, we've noticed that teams adapt to this ambiguity by treating data outputs as conversation starters rather than decision triggers. An alert doesn't mean "act now." It means "let's talk about whether we should act."
A monitoring system detects an anomaly in API response times. Instead of automatically scaling or rolling back the last deploy, it pings a channel. Someone investigates. They confirm the anomaly is real. Then they ask: Should we roll back? Should we scale? Should we wait and see?
The question isn't technical. The metrics already answered the technical question. The question is organizational: who has authority to make this change, under these conditions, without escalating?
In most cases, the answer isn't documented. So the decision defaults to whoever feels senior enough to take responsibility, or it escalates until someone with clear authority makes the call.
The result is that decision-making becomes slower as teams get more data, not faster. More data means more alerts, more dashboards, more ML outputs—and more moments where someone needs to decide whether the data justifies action.
If that decision authority isn't clear, every data point becomes a potential discussion, and every discussion becomes a delay.
The systems behind the systems
The core issue is that most organizations build data systems without designing decision systems. They instrument everything, track every metric, and generate insights at scale. But they don't map those insights to who's authorized to act on them, under what conditions, and with whose approval.
Engineers naturally assume that if the data is clear enough, the decision will be obvious. And technically, it often is. The problem is that obvious decisions still need someone with authority to execute them. And in complex organizations, that authority is almost never as clear as the data.
You can have perfect observability and still spend 30 minutes in a Slack thread debating who should restart a failing service. You can have accurate forecasts and still miss the capacity window because the approval chain doesn't move as fast as the data does.
The systems were designed to provide information. They weren't designed to support decisions. And so teams end up with more data than they can act on, not because they lack insight, but because the organization never clarified who decides what, when.
Top comments (0)