Alerting is a details game. Nobody gets paged by their intent — they get paged by the exact query they wrote. If you want to sleep through the night, the query has to mean what you think it means, and most of the time you find out it doesn't at 3am.
The clearest example I saw wasn't even subtle. A colleague had an alert on message processing count as a throughput signal for our file processor. Perfectly reasonable — if a message were a unit of work. It isn't. One message carries a batch of up to ten URLs, and the thing we actually cared about was PDF downloads.
The important part is that this was not a deliberate approximation. Nobody weighed the cost of instrumenting per-download and settled for messages as a cheaper proxy. The detail that one message holds many URLs simply wasn't known to the person writing the rule, so there was no tradeoff to weigh in the first place. We were tracking the wrong metric outright.
And everything downstream inherited it. The threshold, the SLO, the months of accumulated "is this number normal?" intuition — all of it quietly denominated in the wrong unit.
Five messages, twenty-six units of work, and no constant ratio between them.
That's the shape of every lesson below. The metric is fine. The mapping from metric to intent is where it breaks.
1. Throughput is age-of-oldest-message, not a rate floor
The obvious throughput alert is a floor: fewer than N items in M minutes, page. On a nightly-batch workload this fires every single night, because a floor cannot distinguish "wedged" from "idle, there is genuinely no work."
Age of the oldest queued message has neither problem. If nothing is queued, there's nothing to be old, so quiet periods are silent for free. If work is queued and not moving, age climbs — and it climbs whether the consumer crashed or is up and simply not consuming, which a liveness check misses entirely.
It also spares you from inventing a number. "Is 40 messages/minute normal?" requires a model of your workload. "Has anything been sitting here for six hours?" is grounded in the queue's own drain time, and you can defend it in review.
Same day, same incident. The floor fires seventeen hours out of twenty-four; the age signal fires once.
2. Catching the tail
Queue age tells you that work is not draining. It does not tell you why, and there are two very different whys:
- Is there a system-wide choke, where everything is stalling?
- Or are a handful of genuinely slow jobs dragging out the tail while the main workstream is fine?
A per-unit deadline answers this — "did any single job exceed N minutes?" — and the useful signal is not that it fired, but how many times it fired.
- Many exceedances. You almost certainly got the throughput alert too. This isn't a tail at all: the main workstream is failing, and the tail alert is just a second view of the same outage.
- A few exceedances, throughput healthy. It really is a handful of jobs. You're now looking at specific inputs — one pathological domain, one oversized file — not at the system.
The alert is identical in both rows. The count is what tells you which situation you're in.
Our API is the clean version of the second case. Across a fortnight, about ten requests crossed 10 seconds and everything else was fast. Ten requests out of a fortnight's traffic is not an outage, and a zero-allowance rule at 10s would have paged us for requests that were slow but not broken. So the deadline went where exceedances are genuinely rare — 30 seconds — and the alert now means "something is stuck" rather than "something is slow."
That distinction is the entire reason to run both alerts. Between them, they tell you which of the two questions you're looking at before you open a single dashboard.
3. Ratios need a volume gate, not just a non-zero check
A 99% success SLO sounds like it tolerates failure. On a sparse workload it tolerates nothing: one failure breaches any window holding 100 units or fewer. An hour that saw 3 items and failed 1 reads as 67% and pages you.
We thought this was covered. The SLI carried a no-data guard — the ratio was only evaluated if the service had seen any traffic in the window, so a completely idle hour fell through to "healthy" instead of dividing by zero:
and on() (sum(rate(items_processed_total[1h])) > 0)
That guard does exactly one thing: it excludes the empty window. It says nothing about whether the window holds enough work for a percentage to mean anything. An hour with 4 items clears > 0 comfortably and is still statistically useless.
Over a clean 14-day window with no real incident, one service breached 12 times. Every one of the twelve was an hour holding between 1 and 30 items. Pure denominator artifact.
The fix is to make the guard a volume gate rather than a presence check — the same expression, with a real floor in it:
and on() (sum(rate(items_processed_total[1h])) >= 0.027778) # 100/hour
Zero breaches over the same window, and it still catches every genuine high-volume failure episode. The 99% target never changed. The gate was the whole alert.
Every breach in fourteen clean days, and all of them are hours that barely held any work.
The generalization: any ratio- or percentile-shaped SLI is only meaningful above some volume, and you have to say what that volume is. "Traffic > 0" is not a volume gate.
4. You can only see as far as your largest bucket
Everything right of the dashed line came back as the same number until the buckets were widened.
Everything above assumes you can answer "how many jobs took longer than N?" With a histogram, you can only answer that at a boundary that actually exists. Buckets are cumulative, so "slower than 45 minutes" is count - bucket{le="2700"} — and if nothing ever emitted le=2700, that expression has nothing to subtract.
This is where it turns dangerous. A rule referencing a bucket that doesn't exist does not error. The series is simply absent, the subtraction yields nothing, and the alert quietly never fires. It looks green. It is blind.
The ceiling hides things in the other direction too. Our API's histogram topped out at le=10, so every request slower than ten seconds — 10s, 90s, one wedged request hanging for an hour — landed in the same overflow bucket. We couldn't have set a 30-second deadline if we'd wanted to. Worse, we couldn't even tell whether 30 seconds was the right number, because the data needed to answer that had never been recorded.
So the instrumentation work has to land before the alerting work: push the top boundary above the deadline you care about, with enough headroom to see the shape of what lies beyond it. Ours went from a 5-minute ceiling to 4 hours, which is how we found out that individual scrapes occasionally run for 11.
One catch worth planning for: a bucket added today only describes traffic from today. A new boundary gives you no history, so any threshold derived from it is a guess until you've waited out a full window — and until then, the rule depending on it is one of those green-but-blind rules.
The through-line
Every one of these is the same failure: a plausible metric standing in for the thing you actually meant. Messages for downloads. A percentile for a stuck job. A rate floor for liveness.
So before you ship the rule, replay it against real history and read what it did. Not what it should do — what it did, over a window whose ground truth you already know. It's cheap, and it turns "I think this threshold is right" into "here are the 12 times this would have woken me for nothing."
Top comments (0)