The big observability vendors just admitted out loud what every on-call engineer has known for ten years.
Take a look at Grafana’s rollout of “Adaptive Telemetry” — a feature pitched to slash cloud metrics bills by 30% to 50%. Let’s translate that vendor-speak over a hot cup of chai. The biggest monitoring platform on the planet is now charging you money for pruning shears to throw away the massive pile of useless metrics they spent a decade telling you to ingest. That is not just a polite product launch. That is a full confession.
For years, teams followed the “scrape everything, retain everything, alert on every twitch” playbook. The result? Your monthly cloud bill gives your finance director palpitations, and your on-call pager sounds like an angry cricket every single night. But there is a vendor-neutral, zero-dollar way out of this trap that Google figured out ages ago: Service Level Objectives (SLOs). Not the toothless percentages your VP puts on a quarterly PowerPoint slide, but living, mathematical error budgets that dictate whether your phone actually rings at 3:00 a.m. or stays dead silent on your nightstand.
Here is the no-nonsense playbook: a simple two-question filter to pick clean indicators, the 30-day downtime math nobody teaches in college, copy-paste Prometheus alerting rules that work right away, and a dashboard cleanup rule that will happily delete 150 useless panels from your team’s Grafana folder.
The alert graveyard you already own
Pull up your team’s alert notification channel right now and scroll up. What do you see? A wall of red and yellow noise: CPU usage over 80% on worker-node-04 , Disk space at 85% on cache disk , Pod restarted 3 times , p99 latency (the slowest 1% of all requests) spiked past 400ms. Every single one was created during a post-incident review by someone saying, “Let’s make sure we alert if this ever happens again.”
And every single one of them answers a question nobody on earth cares about in the middle of the night.
When the pager wakes you from deep sleep, there is only one question that justifies turning on the lights: are users actually suffering right now?
None of those infrastructure alerts answer that. A Kubernetes cluster can run its CPU at 94% on a busy festival sale day while customers happily check out in under two seconds. Meanwhile, your payment microservice can throw 500 errors to every single customer while your CPU sits calm and icy at 18%. Infrastructure metrics are just internal weather; they are symptoms, not outcomes. Setting up alerts for raw CPU or memory is like building a hospital where nurses sound an air horn over the PA system every time a patient in ward 4 clears their throat.
Here is what happens in real life: your smoke detector goes off every time someone toasts a piece of bread. Eventually, the human brain adapts. Your engineers don’t suffer from alert fatigue; they develop alert literacy — the dangerous art of hitting “Acknowledge” on Slack without even reading the message. And when the database disk actually corrupts at 4:15 a.m., the alert gets ignored because it sounded identical to the forty false alarms that came before it.
Nobody ever woke up at 3 a.m. grateful their dashboard had 200 panels.
The cure is straightforward: stop alerting on server weather, and start alerting on broken promises. A real engineering promise has three simple gears:
SLI (Service Level Indicator): A clean ratio measuring what the user is experiencing right now.
SLO (Service Level Objective): The reliability target you aim for over a specific window (say, 30 days).
Error Budget: The exact amount of failure you can tolerate before your users get angry and you have to drop everything to fix stability.
If you don’t have all three linked together, you don’t have an SLO. You just have a wish written on a wiki page that nobody reads.
One question runs the whole pipeline: are we inside the budget?
Pick the ratio: SLIs from first principles
Let’s demystify the acronym. An SLI (Service Level Indicator) is nothing more than basic high-school division: good events divided by total events. That is the whole secret. It always lands between 0% (everything is completely down) and 100% (absolute perfection).
Successful HTTP calls divided by total HTTP calls. Requests that finished in under 250 milliseconds divided by total requests. Records processed without errors divided by total batch records.
Before you commit a metric to an alert, run it past these two litmus tests:
1. Does a human user feel it? If an internal Redis cache memory hits 88%, does an end user purchasing a plane ticket feel that? No. If the payment gateway returns a 502 Bad Gateway error, does the user feel it? Instantly. If the user doesn’t feel it, it is a debug metric for an investigation, not an SLI. That single filter eliminates 80% of useless alert spam.
2. Can you express it cleanly as good events over total events? You cannot easily alert on raw percentiles like “p99 latency over 500ms” without falling into complex math traps when traffic fluctuates. Instead, reframe it: “HTTP requests served in under 500ms divided by total HTTP requests.” It captures the exact same customer pain, produces a stable 0-to-1 ratio, and plays nicely with monitoring tools like Prometheus.
There is also a practical engineering consideration: cost to calculate. If measuring an SLI requires spinning up an expensive stream-processing pipeline to parse gigabytes of distributed tracing spans, skip it for now. Use the HTTP status codes your Nginx ingress or API gateway already counts right now. A simple SLI running today beats an architecturally immaculate SLI that ships six months too late.
Depending on what your system actually does, here are the three standard archetypes:
Request-driven systems (APIs, web frontends, mobile backends): Successful responses (non-5xx) divided by total requests; fast responses (under your target threshold) divided by total requests.
Data and batch pipelines (Kafka consumers, nightly ETL jobs): Messages processed before the deadline divided by total messages; records without validation failures divided by total records.
Storage and state systems (Databases, S3 buckets): Successful read/write operations divided by total operations; write acknowledgments confirmed divided by total attempted writes.
Always separate the specification (what users need: “my orders must succeed”) from the implementation (where you grab the counter: gateway access logs vs. Prometheus client libraries). Debate the implementation with your team; never compromise on the user’s view.
Two questions. Fail either and it is a metric, not a promise.
The budget math nobody does
Your SLO is your agreed target — let’s say 99.9% reliability over a rolling 30-day window. Your error budget is simply whatever is left over: 100% minus 99.9% = 0.1%.
Saying “0.1% allowable failure” sounds tiny and academic in a meeting. But when you convert that percentage into cold, hard wall-clock minutes across a 30-day month (43,200 minutes), the reality hits like a splash of cold water:
99% SLO (Two Nines): 1% error budget = 7 hours, 12 minutes of downtime allowed per month.
99.5% SLO: 0.5% error budget = 3 hours, 36 minutes allowed per month.
99.9% SLO (Three Nines): 0.1% error budget = 43.2 minutes allowed per month.
99.95% SLO: 0.05% error budget = 21.6 minutes allowed per month.
99.99% SLO (Four Nines): 0.01% error budget = 4.3 minutes allowed per month.
Treat that table like a monthly prepaid debit card. A 99.9% SLO gives your engineering team exactly 43.2 minutes of total disaster per month to spend. You spend it on database migrations, rolling canary deployments, risky microservice refactors, or unexpected cloud provider hiccups. If you reach day 25 and you still have 30 minutes in the bank, push that new feature! If you blow through your 43 minutes in the first week, the brakes go on: feature releases pause, and the whole team works on resilience.
And let us bury this myth once and for all: 100% reliability is an engineering mistake.
In real-world software, moving code is what causes incidents. If your target is 100%, you can never deploy a security patch, you can never upgrade an operating system, and you can never ship a new feature. You freeze your service into obsolescence. Furthermore, your user is likely sitting on a 4G mobile connection that drops 1% of packets anyway. Paying millions of dollars in redundant infrastructure to deliver 99.999% availability to someone whose home Wi-Fi disconnects when the microwave runs is pure waste. The error budget is not permission to write sloppy code; it is the currency you pay to innovate without paralyzing fear.
An error budget is permission to break things, denominated in minutes. Budget left means ship. Budget gone means freeze and fix.
Illustrative — but count your own alerts per incident sometime.
The only three alerts you need
Now, how do you alert on this? You don’t wait until the entire 43 minutes are gone to sound the alarm; by then, you have already broken your contract with customers. Instead, you alert on the burn rate : the speed at which you are burning through that error allowance relative to standard time.
Think of it like the fuel gauge in a car. A burn rate of 1 means at your current speed, you will run out of fuel right when your 30-day journey finishes. That is completely normal. A burn rate of 14.4 means you are stomping the gas pedal so hard you will drain the entire month’s tank in just 50 hours!
The Google SRE workbook establishes the gold standard for alerting against a 30-day window. Instead of managing 50 different flapping alerts, you need just three mathematical rules:
Burn Rate 14.4 (Consuming 2% of your monthly budget in 1 hour): PAGE A HUMAN. A major chunk of your monthly allowance is vaporizing right now. Pick up the phone immediately.
Burn Rate 6 (Consuming 5% of your monthly budget in 6 hours): PAGE A HUMAN. A slower bleed, but serious enough that you will breach your SLO before the day ends.
Burn Rate 1 (Consuming 10% of your monthly budget in 3 days): FILE A TICKET. A minor background leak — maybe a single broken endpoint used by a niche customer. Do not wake anyone up at 2:00 a.m. Log an engineering ticket for normal working hours.
To avoid false alarms from brief 30-second network blips, we use multiwindow multi-burn-rate alerting. Every alert checks two things at once: a long window (to ensure the burn is genuinely dangerous) and a short window (one-twelfth the duration, to ensure the issue is still ongoing and hasn’t already self-healed). If a transient hiccup clears up after 40 seconds, the short window resets, and the pager never wakes you up.
Here is how you write this cleanly in Prometheus (the standard open-source monitoring tool used across Kubernetes and cloud infrastructure):
groups:
- name: slo-burn-rate
interval: 30s
rules:
# Error ratios over long and short windows.
# Swap the matchers for your service's labels.
- record: job:slo_errors:ratio_rate1h
expr: |
sum(rate(http_requests_total{status=~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
- record: job:slo_errors:ratio_rate5m
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
- record: job:slo_errors:ratio_rate6h
expr: |
sum(rate(http_requests_total{status=~"5.."}[6h]))
/
sum(rate(http_requests_total[6h]))
- record: job:slo_errors:ratio_rate30m
expr: |
sum(rate(http_requests_total{status=~"5.."}[30m]))
/
sum(rate(http_requests_total[30m]))
These recording rules pre-calculate your error ratio so your Prometheus server doesn’t choke running heavy calculations during a crisis. Now, here are the alerting rules that consume them:
groups:
- name: slo-alerts
rules:
# 2% of a 30-day budget in 1h -> burn rate 14.4 - PAGE
- alert: ErrorBudgetBurnFast
expr: |
job:slo_errors:ratio_rate1h > 14.4 * 0.001
and
job:slo_errors:ratio_rate5m > 14.4 * 0.001
for: 2m
labels:
severity: page
annotations:
summary: "Fast error-budget burn: 2% in 1h"
# 5% of a 30-day budget in 6h -> burn rate 6 - PAGE
- alert: ErrorBudgetBurnSlow
expr: |
job:slo_errors:ratio_rate6h > 6 * 0.001
and
job:slo_errors:ratio_rate30m > 6 * 0.001
for: 15m
labels:
severity: page
annotations:
summary: "Slow error-budget burn: 5% in 6h"
Where does that 0.001 come from? That is the 0.1% allowable failure for a 99.9% SLO expressed as a decimal ratio. If your service targets 99.95%, your budget is 0.05%, so you simply change that constant to 0.0005. Everything else remains pure, predictable math.
Start with these two paging rules. Once your team trusts them and realizes the pager only screams when users are genuinely impacted, add the 3-day ticket rule to clean up minor background leaks during your weekly sprint.
Let the SLO drive the dashboard
Once your alerts are anchored to user experience, open up your Grafana dashboard folder. It’s time for an audit.
Most engineering dashboards look like the flight deck of an Airbus 380: 250 panels crammed with CPU spikes, GC pauses, JVM thread pools, and TCP socket retransmits. When an incident occurs, three engineers stare at it and argue about whether a blue line wobbling at the bottom right has anything to do with why checkout is failing.
Organize your dashboard into three ruthlessly prioritized tiers:
Top Left (The Anchor): A single giant dial showing two numbers: Error Budget Remaining (e.g., “78% remaining”) and Current Burn Rate (e.g., “0.8x”). Anyone from a junior intern to the CTO can glance at this panel and instantly know if the service is healthy.
Second Row (The Experience): The SLI graphs over the rolling window — availability ratio and latency ratio plotted against your target threshold.
Below the Fold (The Diagnostic Graveyard): Move your CPU, disk, memory, and thread metrics down here. They are debugging tools, not health indicators. If a metric panel cannot directly help you diagnose why the budget is burning, delete it without remorse.
Pruning those dashboards might make you feel uneasy at first. That discomfort is just the old “alert-on-everything” habit slowly leaving your system.
Your 7-day action plan to get your first SLO live:
Select one critical service and identify the exact end-user action you want to protect (e.g., adding an item to the shopping cart).
Define your SLI as a clean ratio: successful user actions divided by total attempted user actions.
Pick a reasonable target to start: 99.9% over a 30-day window is a reliable default.
Calculate your downtime allowance in actual minutes (43.2 minutes for 99.9%) and pin it visibly to your team’s board.
Deploy the Prometheus recording rules and the two paging burn-rate alerts shown above.
Put the error budget dial right at the top-left of your team dashboard; demote infrastructure noise below the fold.
Make an agreement with product management: when the error budget hits zero, new feature deploys pause until reliability is restored.
Keep this handy. It replaces dozens of messy internal runbooks.
The takeaway
Traditional monitoring tools were built to tell you what is happening under the hood. SLOs ask a much smarter question: does anyone actually care?
When you let the error budget handle triage, your team stops guessing what matters at 3:00 a.m. based on tired intuition. Keep these three core principles close:
Promises beat symptoms: Alert on user-facing ratios, never on server weather.
The budget controls the pager: Three burn-rate rules (14.4, 6, and 1) comfortably replace fifty flapping threshold alerts.
Dashboards serve the customer: If a visual graph doesn’t explain why your error budget is draining, it is just decorative noise.
Pick one service this week, write down its ratio, and reclaim your nights. Observability vendors will always try to sell you expensive pruning tools. But now, you know how to tend your own garden.
Found this helpful? Pass it on to the engineer taking over the on-call pager this Friday. Sharing keeps this publication growing — and helps keep all our pagers quiet.
So here's my question for you: if you wrote one service's user-facing SLI as a simple ratio today, what would the numerator and denominator be?
Disclaimer
This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.
© 2026 Honey Kiran. All rights reserved.
Honey Kiran · DevOps & Platform Engineering
Practical guides on monitoring, SRE, Terraform, and platform engineering for high-scale systems.
Your turn: Take one service this sprint and draft its user-facing SLI as a simple ratio. Leave a comment or reach out with your candidate — we’ll review the trickiest edge cases in a follow-up teardown.
Examples are illustrative — validate in your own environment before running in production.
Top comments (0)