DEV Community: P Bhatnagar

The Monitor That Only Passed When It Won a Race

P Bhatnagar — Sun, 27 Sep 2026 10:06:48 +0000

A browser monitor I was migrating failed about two runs in every three. The rest passed cleanly. The application behind it was healthy. Nobody could see a pattern.

The run history for that stretch was 67 runs. 25 passed. 42 failed. That is a 37.3% pass rate on a monitor watching an application that was working.

This post is about how that failure was found and fixed. The fix itself was small. The idea behind it is the part worth keeping:

A check that passes only when it wins a race is not a passing check.

Why intermittent failures are the hardest kind

A monitor that always fails is easy. You look at it, find the broken thing, and fix it.

A monitor that fails sometimes is harder, for three reasons.

It teaches people to ignore it. After a week of flapping, a red run stops meaning anything. That is the same noise problem I wrote about in Cutting Alert Noise Without Going Blind. A flapping monitor is noise with a green light mixed in.

It passes on a rerun. So the easy "fix" is to add a retry and move on. The dashboard goes green. The race is still there.

There is no single broken thing to look at. The passing runs and the failing runs use the same script against the same application. Comparing them line by line shows nothing.

So the useful question is not "what is broken?" It is "what is different between a run that passes and a run that fails?"

Reading the pattern

Two things stood out in the step results.

First, the failures happened at the same step: sign in. The monitor signs in through a single sign on page before it reaches the application. That is where the failing runs stopped.

Second, a run could fail and the next one could pass with nothing changed in between.

Same step. Random outcome. Passes on a rerun. When I see that combination, I treat it as a timing problem until something proves otherwise. The script and the page are not waiting for each other properly.

In The Health Check That Was Red and Green at the Same Time, I wrote about reading failure duration first. A very fast failure points at the environment: network, DNS, routing. A failure that runs its course and then fails a check points at the script side: a selector, an assertion, or sign in. This one belonged on the script side. The application was fine. The script's timing was not.

The race

After you enter credentials, the identity provider does some work before it hands you back to the application. Along the way it can show an optional prompt asking if you want to stay signed in. How long all of this takes is not fixed. It changes from run to run.

The script did not account for that. It used a fixed wait after sign in, then moved on as if the next page would be there.

When the sign in flow settled inside that wait, the run passed. When it settled later, the script acted on a page that was not ready yet, and the step failed.

This changes how you read the 25 green runs. They were not proof that the script worked. They were runs where the timing happened to line up. The greens were luck, not health. A monitor like that is not telling you anything about the application. It is telling you about a race it sometimes wins.

Why the obvious fixes are wrong

There are two quick ways to make this monitor go green. Both are traps.

Make the wait longer. This makes every run slower, even the ones that were ready early. And it is still a guess. The day the identity provider has a slow afternoon, the race comes back.

Add retries. This hides the flap from the dashboard. It also hides a real sign in outage. If sign in is genuinely broken for a while, a retrying monitor turns a clear red into a slow, confusing mix of passes and failures. You find out later than you should.

Both fixes change what the dashboard shows. Neither changes what the script actually knows. That is the same rule I keep coming back to: never loosen a check just to make a dashboard green.

The fix: poll and proceed

The real fix is to stop guessing a duration and wait for a condition instead.

Instead of "wait N seconds, then continue," the script asks a question again and again, for a limited time:

Is the landing page there? Then sign in is done. Move on.
Is the stay signed in prompt there? Then answer it, and keep checking.
Neither yet? Wait briefly and check again.
Past the time limit? Fail, and say exactly why.

Here is the shape of it in simplified pseudocode. This is not the monitoring tool's syntax, and it is not the real script. It is just the logic.

Before: wait a fixed time and hope.

submit_credentials()
sleep(FIXED_WAIT)          # a guess about the identity provider
continue_to_application()  # assumes the page is ready

After: poll for a condition, with a limit.

submit_credentials()

deadline = now() + TIME_LIMIT
loop:
    if landing_page_visible():
        break                          # signed in, move on
    if stay_signed_in_prompt_visible():
        answer_prompt()                # optional: handle it only if it shows
    if now() > deadline:
        fail("sign in did not complete within TIME_LIMIT")
    sleep(SHORT_POLL)

continue_to_application()

Three things make this version better.

It waits for the page, not the clock. Fast runs move on as soon as they are ready. Slow runs get the time they need.
It handles the optional prompt either way. The prompt can show or not show. The script no longer cares which.
It is still bounded. If sign in never completes, the step fails with a clear reason. A real sign in outage still shows up as red. That is the part retries break.

What the numbers support, and what they do not

This is the section I care about most, because it is where it is easiest to overclaim.

The first two runs after the fix passed. Two runs prove nothing on their own, so I did not call it done there.

The longer window in nonproduction after the fix was 256 passes out of 333 runs, or 76.9%. On its own, that looks like the fix only half worked. It did not mean that. This application was scoped for production monitoring, and the failures that remained in that window were environmental, not the sign in race. The monitor was later paused in nonproduction, because monitoring for that application lives in production.

In production, the monitor is at 100% on the current sample, with a P90 of about 58 seconds.

So the honest claim is: the flap is gone, and the application runs healthy in production. I do not say "100% after the patch." The full window does not support that sentence, and a number you cannot defend is worse than a smaller number you can.

A second sign in failure, with a different shape

In the same migration, another production application also failed at authentication. It looked similar at first. It was not.

This one did not flap. It failed from our private runner, while the same check in the legacy system showed the application healthy at 100%. A consistent failure that is healthy at the source does not point at timing. It points at the environment between the runner and the application.

I did not have access to the cluster that hosted that private location. So I designed a diagnostic and asked the engineer who owned it to run it: open a shell inside the runner pod and call the endpoint directly from there. That removes the monitoring script from the picture completely.

The result was clear. The TLS connection succeeded. The response was HTTP 401 with a message that a bearer token was required. It was not a firewall block and not a script defect. The API gateway was enforcing OAuth on the runner's path. The blocker went to the gateway team with that evidence attached, and the test was left as it was.

Both failures happened at sign in. What decided the next move was not the step they failed on. It was the shape of the failure:

Intermittent at the same step means timing. Fix the script.
Consistent, but healthy at the source means environment. Prove it, escalate with evidence, and do not touch the test.

One small note on diagnosing through someone else's hands. When you cannot run a check yourself, make it as small as possible and decide in advance what each possible result would tell you. Then one run by another person is enough to settle the question.

Takeaways

Intermittent means timing until proven otherwise. Same step, random outcome, passes on a rerun: that is the signature.
A check that passes only when it wins a race is not a passing check. Treat its green runs as luck.
Wait for conditions, not durations. And put a limit on every wait, so a real outage still fails loudly.
Do not fix a flap with longer sleeps or retries. Both hide the race instead of removing it.
Let the shape of a failure pick the next step. Intermittent points at the script. Consistent and healthy at the source points at the environment.
Claim what the numbers support. "The flap is gone and it is healthy in production" is a smaller sentence than "100% after the patch." It is also the one that survives a follow up question.

This is the fourth post in a series on the practice of running monitoring. The earlier ones cover alert noise, a health check that was red and green at once, and shipping monitoring changes without causing an incident.

Zero Blast Radius: Shipping Monitoring Changes Without Causing an Incident

P Bhatnagar — Thu, 06 Aug 2026 05:15:57 +0000

The first time I saw a monitoring change cause a problem, nothing was actually broken.

We were building out dashboards and alerts for a set of production applications. Building an alert meant configuring its notification path, and the notification path pointed at the application team that owned the service: their distribution list, their paging rotation. The alerts were still under test. The thresholds had not been agreed with anyone. Nobody had signed off on receiving them, or on what they were supposed to do when one arrived.

They received them anyway. People were pulled toward something that was not an incident, for an alert they had never agreed to own, at a threshold nobody had defended yet.

Nothing was down. It still cost hours, and it cost something slower to earn back, which is that team's willingness to treat the next alert from us as real. The fix was small and it became a standing rule for the work that followed: recipients go in last, after signoff, not while you are still deciding whether the alert is any good.

That rule is the narrow version of a wider one. Monitoring changes get treated as low risk, because they are not production code and they do not touch customer traffic. A detector with a badly chosen threshold pages an on-call engineer at 3 AM for something nobody can act on. A monitor pointed at the wrong endpoint reports a healthy service as down, and someone starts a bridge call. A bulk write against a monitoring API overwrites an alert routing rule that took two teams a week to agree on. None of that is an application outage, and every one of them burns trust in the alerting layer, which is the thing you were trying to improve.

I later spent a long stretch moving a synthetic monitoring estate from a legacy platform to a new one, then building detectors on top of it across production applications. Most of that work was done through APIs, in a live environment, with credentials scoped to production. So I settled on a routine built around one rule. A monitoring change should never be the thing that causes an outage. Its job is to report one.

The checklist

✅ Create paused, with no recipients. The object exists, it cannot fire, and nobody is wired to it.
✅ Verify by hand. Run the check, read what it actually asserts on.
✅ Backtest. Run the condition against history before it runs against now.
✅ Approve activation separately. Not the same change request that created it.
✅ Enable with the rollback ready. One object at a time.
✅ Reconcile. Nothing gets left paused by accident.

On paper the sequence is straightforward. The return paths are where most of the effort goes:

Create in the off position

Every monitor is created paused. Every detector is created disabled. And the recipient list is the last field you populate, not the first. Creation and activation are two different decisions, and combining them removes your only chance to inspect the thing before it can act. When a detector is created live, the first evidence you get about whether its threshold is sane is a page, and it is somebody else's page.

Most platforms support pausing directly. Where they do not, an alert created with no recipients at all is the fallback. That is weaker than a real disabled state, because a half-configured object in production is its own hazard, but it beats going live blind. The important part is not the API flag. It is that you have separated "the thing exists" from "the thing can wake someone up."

Verify before anyone sees it

Once the object exists in the off position, run it manually and read the result properly.

For a synthetic monitor that means executing the check against production and looking at the run in detail: did it complete, how long did it take, what did it actually assert on, and does the pass reflect the service being healthy or the assertion being too loose. A monitor that passes for the wrong reason is worse than one that fails honestly, because it will keep passing during the outage you built it for.

For a detector or alert rule, it means running the condition against history before it is allowed to run against the present. Some platforms give you this directly: as you configure the rule, the preview chart applies your condition to the stored signal and shows the windows where it would have entered the alerting state. Widen that preview to two or three weeks and count them. Where the platform has no preview, you can do the same thing by hand, by taking the rule's own query, bucketing it to the rule's evaluation interval, applying the same trigger condition, and counting what survives. The bucket has to match the interval. A rule that evaluates every five minutes, checked against hourly buckets, produces a number that means nothing.

Two things make the count readable. First, separate crossings from episodes: a condition that flips true, false, true across twenty minutes is one event and can produce eight notifications. Second, a count of zero is only informative if the window you checked contained an incident you can name. Over a quiet fortnight, zero tells you the fortnight was quiet.

Here's a manual backtest for an alert that fires when the HTTP 500 rate exceeds 5% in any five-minute window, run over the last three weeks:

The exact syntax varies by platform. The example below uses Splunk SPL because it makes the backtesting approach explicit, but the same idea applies regardless of your monitoring stack.

index=web_logs sourcetype=access_combined
| bin _time span=5m
| stats count(eval(status>=500)) as errors count as total by _time
| eval error_rate=if(total>0,round(errors*100/total,2),0)
| where error_rate>5
| stats count as breaching_windows

That counts breaching windows. To estimate distinct alert episodes instead, treat breaching windows separated by less than fifteen minutes as part of the same event:

index=web_logs sourcetype=access_combined
| bin _time span=5m
| stats count(eval(status>=500)) as errors count as total by _time
| eval error_rate=if(total>0,round(errors*100/total,2),0)
| where error_rate>5
| sort 0 _time
| streamstats current=f last(_time) as prev_time
| eval episode_start=if(isnull(prev_time) OR (_time-prev_time)>900,1,0)
| stats sum(episode_start) as episodes count as breaching_windows

The gap between those two numbers tells you which knob to turn. Sixty breaching windows across four episodes means the threshold is roughly right, but the rule probably needs a duration clause before it notifies. Sixty breaching windows across fifty episodes usually means the threshold itself is too sensitive.

Either way, this is the highest-value step in the sequence because it is the only one that tests the rule against reality rather than against your intent.

Activation needs its own approval

The change request that let you create the object should not be the change request that lets it fire.

At first glance this looks like process for its own sake. In practice it does something specific: it forces the person approving activation to look at evidence that did not exist when the work was first proposed. At creation time all anyone can review is your intent. At activation time there is a real object, a real manual run, and a real backtest. The second conversation is short, because you are handing over evidence rather than a plan.

It also gives you a clean place to stop. If the backtest looks wrong, nothing goes live and nothing needs reverting, because the object was never able to do anything.

Enable, one at a time

Enable one object and stay with it. Not the batch.

The first evaluation cycles after activation are where you find out whether the backtest matched reality, and you only get that signal if you are still watching. Enable twenty at once and the first firing arrives with no way to tell which change produced it, which is the position you spent the previous four steps avoiding.

Treat that first firing as information about the threshold rather than as a result. A detector that fires within minutes of going live is more often telling you something about your condition than about the service. Check the underlying signal before anyone acts on it. If the threshold turns out to be wrong, nothing has reached a human yet and the rollback line is already written.

GET before you write, always

Every write against a production monitoring API is preceded by a read of the current state of that same object, and the response is saved to a timestamped file.

This costs one extra call. Monitoring APIs are frequently full-replacement rather than partial-update, so a PUT built from a partially populated payload will quietly drop fields you never intended to touch: notification rules, tags, team assignments, custom properties. You will not see this in the response, because the write succeeds. You see it a week later when an alert routes to the wrong team.

mkdir -p backups
STAMP=$(date -u +%Y%m%dT%H%M%SZ)

curl -sS -H "$AUTH_HEADER" "$API/detector/$ID" \
  | tee "backups/detector-${ID}-${STAMP}.json" \
  | jq -e '.id' > /dev/null

Two details in there matter. The object ID plus a UTC timestamp in the filename means that during a rollback you are not guessing which of four exports was the pre-change one. And the check on the end means an auth failure or an empty body cannot masquerade as a successful backup. A zero byte file in the backups directory is worse than no file, because you will trust it.

For bulk operations, export everything in scope first, as a single operation, before the first write goes out. Do not export as you go. If the script fails halfway through, the objects you have not reached yet are the ones you most need originals for.

Write the rollback first

The rollback for any monitoring change should fit on a single command, and it goes into the change record before the change is applied, with the real filename in it rather than a placeholder.

ID=abc123
BACKUP=backups/detector-${ID}-20260805T181200Z.json

curl -sS -X PUT "$API/detector/$ID" \
  -H "$AUTH_HEADER" \
  -H "Content-Type: application/json" \
  --data @"$BACKUP"

Writing it afterwards defeats the point. The moment you need it is the moment you are least able to compose it, so the discipline is in having decided in advance what the undo is, not in the complexity of the command. For most of this work the undo genuinely is one line: disable the detector, pause the monitor, or PUT back the saved export.

If you cannot express the rollback in one line, that is useful information about the change. It usually means you are changing several objects at once and should be splitting the batch.

Scope the credentials

Use a token scoped to the specific environment and, where the platform allows it, to the specific capability. An org-wide admin token used for a routine dashboard update is a blast radius you are carrying around for no benefit.

This one is easy to skip because the wide token already exists and already works. The argument for the narrow one is not really about malice. It is about the script that has a bad loop bound, or the copy-pasted object ID from the wrong environment. A narrowly scoped token turns those into an error message instead of an incident.

The failure mode this creates

Every safety pattern introduces its own way of going wrong, and being honest about that is more useful than presenting the pattern as free.

Creating things in the off position means you can forget to turn them on. A monitor built during a migration, verified, approved, and then left paused because the activation window slipped, is invisible coverage debt. Nobody notices, because a paused monitor never complains. You find out during the incident it would have caught.

The fix is a reconciliation step, not a better memory. At the end of every change window, list every object in the paused or disabled state and account for each one: intentionally off, or dropped. Anything older than the window that nobody can explain gets either enabled or deleted. Most monitoring APIs will give you that list in one call, so this is a five minute task that closes the only real hole in the pattern.

I would rather carry that failure mode than the alternative one. Forgetting to enable a monitor is a gap you can find with a query. Enabling a bad detector is a gap that finds you.

What it actually costs

Yes, it is slower. Two approvals instead of one, an export step before every write, and a backtest that occasionally sends you back to redesign a threshold you thought was finished.

What you get in exchange is that the monitoring layer stops being a source of incidents. Across the migration and the detector rollout that followed, just over forty monitor definitions and twenty-six detectors went live across sixteen production applications. Nothing I enabled had to be rolled back for firing wrongly, and no change I made took out an existing alert path. That is not because the thresholds were inspired. It is because every one of them was inspected while it was still incapable of paging anyone.

Monitoring exists to reduce operational risk. When deploying it adds risk instead, the thing that needs fixing is the deployment process rather than the monitoring. The same pattern applies well beyond monitoring, and most infrastructure work has some version of the off position available. It is worth finding out what yours is before you need it.

If you run a variant of this, particularly the reconciliation step, I would like to hear how you handle it.

The Health Check That Was Red and Green at the Same Time

P Bhatnagar — Mon, 27 Jul 2026 15:05:31 +0000

A business-critical health check was showing red. Red on the dashboard, red in the alert history, red enough that someone was about to get paged. But when I opened the failing runs, the service underneath was returning HTTP/2 200. The check was failing, and the thing it was checking was fine. Red and green at the same time.

That contradiction turned into one of the more useful debugging lessons I've had in monitoring, and the thing that cracked it wasn't a better tool or a deeper log dive. It was a single habit: read the failure duration before anything else.

Read the duration first

When a synthetic monitor fails, the first number I look at isn't the error message. It's how long the check ran before it failed.

Fast fail (under roughly 150 milliseconds, with no response body read): the check died before it ever got a real answer from the application. That points at the environment: network, DNS, routing, egress, a blocked runner. The application was never actually reached.
Full duration, then an assertion failure: the check ran the whole transaction, got a response, and then decided that response was wrong. That points at the script: a selector, an SSO step, or the assertion logic itself.

This one split does most of your triage before you read a single log line. The duration tells you which half of the world to look in: the environment the test runs in, or the test itself.

Back to the red-and-green health check. It was failing in about 24 milliseconds. No real round trip completes in 24 milliseconds, and the response headers showed HTTP/2 200 underneath. Fast fail plus a healthy status: by the heuristic, this wasn't the application failing. It was the assertion.

The assertion that lied

The check had exactly one assertion: pass only if the status code equals 200.

That looks sensible until you notice how much it depends on. The status code has to survive the whole path from the service, through whatever proxies and gateways sit in between, to the bit of code inside the check that reads it and compares it. Anywhere along that path it can arrive in a form the check doesn't recognise, and when that happens the check reports a failure even though the service answered correctly. That's what was happening here. The response headers said HTTP/2 200. The assertion still called it red.

A status-code-only assertion is weak in both directions, and the direction nobody notices is the dangerous one. The false red is loud. Someone gets paged, someone investigates, someone eventually finds out the application was healthy the whole time. The false green is silent. Gateways, CDNs, and friendly error pages will happily hand you an HTTP 200 wrapped around an empty body or an error message, and a check that only reads the status code will call that a pass. Your dashboard stays green through a real outage.

The fix was to make the assertion ask a question worth asking. Instead of checking the status code, check that the response body actually contains what a healthy response contains, a specific content marker that only a genuinely working service returns, and pair that with a duration sanity window. The duration window catches both ends: a hang, and a suspiciously instant response that couldn't possibly have done any real work. Status code stays in as a supporting signal, never as the sole criterion.

The rewritten check went green immediately and stayed there. Three hundred consecutive passes, median around 23 milliseconds.

The tooling copied the mistake faithfully

That bad assertion didn't come from nowhere. It came from an old test template, and the conversion tooling that migrated the legacy scripts had reproduced it exactly, everywhere the template had been used. This is worth sitting with if you're using AI-assisted conversion for any kind of migration. The tooling did precisely what it was asked to do. It was faithful. It just had no opinion about whether the thing it was faithfully reproducing was any good, so a single bad pattern from years ago got copied forward at machine speed into a brand new platform.

Assertions are the part of a migrated test that most needs a human to read it, because an assertion is the only place where the test states what "working" means. Everything else is mechanics.

But going green was never the point. The point was making sure it couldn't come back. I generalized the fix into a platform rule, no status-code-only assertions anywhere, and replicated the corrected pattern to the sibling monitors that had inherited the same template. The goal was to make the mistake structurally unrepeatable, not just fixed in one spot.

The discipline that matters most

There's a version of this job where every red monitor is an annoyance to be silenced, and the fastest way to silence one is to loosen its assertion until it passes. Resist that completely.

Not long after, two monitors started failing together while their sibling checks in other environments passed the same set of test cases. The pattern was suspicious in a specific way, same environment, same data lookups, and it would have been trivial to relax the assertions and make the red go away. I didn't, because the evidence pointed at a genuine data-layer problem in that environment. Loosening the assertion wouldn't have fixed anything. It would have hidden a real "data not found" behind a green light. The right move was to keep the assertion honest and escalate the underlying cause.

That's the rule I'd hand a new monitoring engineer on day one: if a check fails, diagnose the root cause, and never adjust the assertion just to make the test pass. The moment you loosen an assertion to force green, you've converted your monitor from something that tells you the truth into something that tells you what you want to hear.

What it all comes down to

A synthetic monitor has exactly one job: tell you the truth about what a real user would experience. Everything here is in service of that one job.

Read the failure duration first. It tells you whether to suspect the environment or the test.
Assert on meaning, not on a status code. Verify the response actually contains what a healthy response contains.
Never loosen an assertion to force a green. Diagnose the cause instead.
Review anything your automation generates. It reproduces your bad assumptions as faithfully as your good ones.

The monitor that was red and green at the same time wasn't broken hardware or a flaky network. It was a monitor asking the wrong question. Once it asked the right one, it started telling the truth, and a monitor that tells the truth is the only kind worth having.

Cutting Alert Noise Without Going Blind: A Practical, SLO-Driven Approach

P Bhatnagar — Wed, 22 Jul 2026 14:46:21 +0000

There's a specific kind of failure that keeps me up at night, and it isn't the outage. It's the outage that sat unseen because the alert announcing it was buried under two hundred others that didn't matter.

Most teams treat alert fatigue as an annoyance: a noisy channel, a phone that buzzes too much. It isn't an annoyance. It's a reliability risk. Every alert that fires without needing a human response is a small withdrawal from the same account: the team's trust that a page means something. Drain that account far enough and people start muting, skimming, and reflexively acknowledging without looking. At that point your monitoring is still running, still green on the dashboard, and functionally blind. The danger was never the noise itself. It's what the noise trains people to ignore.

After years of keeping production systems observable, I've come to think the goal of good alerting isn't fewer alerts for their own sake. It's a team that trusts every page, because every page is real. Here's how I get there.

Why noise accumulates

Noise is rarely one bad decision. It's the compound interest of small, reasonable-at-the-time choices that nobody revisited. A few patterns show up almost everywhere:

Static thresholds that outlived their context. Someone set "alert if latency > 800ms" during a launch two years ago. Traffic patterns changed, the baseline moved, and the threshold quietly became either a constant false alarm or a rubber stamp that never fires. Thresholds decay. Most teams never schedule the review.

Alerting on causes instead of symptoms. CPU at 90% pages someone. But 90% CPU is not a problem. It's a machine doing work. The problem is a user getting an error or waiting too long. When you alert on every internal cause, you generate a page for every twitch in the system, most of which resolve themselves before anyone reads the notification.

No severity tiering, so everything pages. When the informational, the interesting, and the genuinely urgent all arrive through the same channel with the same urgency, the person on call has to sort them out in their own head, doing the triage the alerting system should have done for them. That cognitive tax is where fatigue actually lives.

One incident wearing three costumes. A single upstream failure lights up the load balancer, the application layer, and the database monitor as three independent alerts. The responder now spends their first five minutes proving to themselves that it's one incident, not three, during the exact window when minutes matter most.

None of these come from carelessness. They come from adding monitoring faster than anyone prunes it.

The reframe: alert on symptoms, not on every metric

The shift that changes everything is deceptively small. Stop asking "is this metric abnormal?" and start asking "is a user being hurt right now, and can a human do something about it?"

That second question is a much higher bar, and it's the right one. It moves your alerting off the internals of the system and onto the experience of the people using it. An abnormal metric is a curiosity. A hurt user with an actionable cause is a page.

This is where Service Level Objectives earn their keep. When you define what "working" means from the user's side (a success rate, a latency ceiling, an availability target), you get a principled way to decide what deserves human attention. You're no longer alerting on raw metrics; you're alerting on whether you're spending your error budget faster than you can afford to. A slow burn against your budget is a ticket to look at tomorrow. A fast burn that will exhaust the month's budget in an hour is a page tonight. Same underlying data, completely different urgency, and the difference is decided by user impact, not by a number crossing a line.

A practical example

Consider a checkout service. A traditional alert might look like this:

Alert: CheckoutHighErrorRate
Condition:
  http_requests_total{status=~"5.."} > 100
Duration: 5m
Severity: Critical

The problem is that raw error counts say nothing about customer impact. One customer stuck in a retry loop can generate hundreds of errors. Fifty different customers each failing once is a much bigger business problem, and this alert can't tell the two apart.

A better alert ties together user impact and SLO burn:

Alert: CheckoutSLOBurn
Condition:
  burn_rate > 2
  AND affected_customers > 50
  AND duration > 10m
Severity: Critical

Same system. Very different signal. The first alert asks, "did errors increase?" The second asks, "are enough real users being hurt that someone should wake up?"

From metrics to action

Raw signals aren't the thing you page on. They're the thing you refine, through correlation and user-impact and burn rate, until what's left is worth a human's sleep.

A framework you can apply Monday

Principles are easy to nod at. Here's the operational version: four moves you can start on this week.

1. The actionability test. For every alert that pages a human, answer one question: what specific action does the person take when they receive it? If you can name the action (restart this, fail over to that, roll back the deploy), it's a legitimate page. If the honest answer is "check whether it's still happening," it isn't a page. It's a dashboard panel or, at most, a ticket. This single test retires more noise than any other. Run it against your current paging alerts and you'll be surprised how many fail.

2. Tie severity to impact, not to the metric. Build three lanes and route deliberately. Page for user-facing harm with an actionable cause and meaningful error-budget burn. Ticket for things that need attention but not tonight: degradations, slow trends, capacity you'll hit next quarter. Silent dashboard for the informational signal you want visible but never want to be woken by. The discipline is refusing to let a lane creep upward. Most alert bloat is ticket-worthy signal that somehow got wired to the pager.

3. Deduplicate and group. Collapse correlated alerts before they reach a person. If a dependency is down, suppress the downstream alerts that are merely reporting the same root failure. Dependency-aware suppression turns three costumes back into one incident. Respect maintenance windows so planned work doesn't page anyone. The responder should receive the incident, not the incident's echoes across every layer it touched.

4. Enrich with context. An alert that says "error rate high" makes the responder start from zero. An alert that arrives with the runbook link, the last few deploys, and a clear statement of what is failing and why we think so lets them start from the middle. Attach the filtering and correlation logic that already knows which paths are affected and which are noise. The responder's first job should be deciding what to do, not reconstructing what happened.

A quick decision tree

If every alert had to walk this path before reaching a phone, most of them wouldn't make it, and that's the point.

Knowing it's working

You can't manage what you don't measure, and "the channel feels quieter" is not a metric. Track the ones that are:

Metric	Why it matters
Alert volume	Measures overall noise
Actionable pages	Indicates trust in the system
Auto-resolved alerts	Shows suppression opportunities
False positives	Direct measure of alert quality
MTTR	Measures operational improvement

If I had to pick one, it's this: measure the reduction in pages that led to no action. That number is the purest expression of the whole effort. Driving it down means you're returning human attention to the system, one retired alert at a time.

One caution before you automate

Everything above can be automated: generated thresholds, auto-suppression rules, ML-driven alerting. You should automate it. But keep one thing in mind, because it's the trap I see teams fall into most. Automation amplifies whatever judgment you feed it. An auto-suppression rule will silence a noisy alert, and it will just as faithfully silence a real failure that happens to match the same pattern. A generated threshold propagates your best thinking and your worst assumptions with equal enthusiasm, and the bad ones scale faster because nobody's watching. The answer isn't to stop automating. It's to automate the execution and keep a human review gate on the judgment.

The takeaway

If you do nothing else, do these three things this week:

Run the actionability test against your current paging alerts. Demote every one that has no concrete human action attached.
Pick your single noisiest alert and either retire it, re-tier it, or enrich it so it's worth reading.
Start tracking pages that led to no action. That one number tells you whether any of this is working.

Final thoughts

Every mature monitoring platform eventually collects thousands of alerts. The best teams aren't the ones that monitor the most. They're the ones that continually remove everything that no longer deserves a human's attention.

That's the real job: not building alerts, but building trust. Because at three in the morning, nobody is reading every notification. They're betting that the one which woke them up is worth believing.