The first time I saw a monitoring change cause a problem, nothing was actually broken.
We were building out dashboards and alerts for a set of production applications. Building an alert meant configuring its notification path, and the notification path pointed at the application team that owned the service: their distribution list, their paging rotation. The alerts were still under test. The thresholds had not been agreed with anyone. Nobody had signed off on receiving them, or on what they were supposed to do when one arrived.
They received them anyway. People were pulled toward something that was not an incident, for an alert they had never agreed to own, at a threshold nobody had defended yet.
Nothing was down. It still cost hours, and it cost something slower to earn back, which is that team's willingness to treat the next alert from us as real. The fix was small and it became a standing rule for the work that followed: recipients go in last, after signoff, not while you are still deciding whether the alert is any good.
That rule is the narrow version of a wider one. Monitoring changes get treated as low risk, because they are not production code and they do not touch customer traffic. A detector with a badly chosen threshold pages an on-call engineer at 3 AM for something nobody can act on. A monitor pointed at the wrong endpoint reports a healthy service as down, and someone starts a bridge call. A bulk write against a monitoring API overwrites an alert routing rule that took two teams a week to agree on. None of that is an application outage, and every one of them burns trust in the alerting layer, which is the thing you were trying to improve.
I later spent a long stretch moving a synthetic monitoring estate from a legacy platform to a new one, then building detectors on top of it across production applications. Most of that work was done through APIs, in a live environment, with credentials scoped to production. So I settled on a routine built around one rule. A monitoring change should never be the thing that causes an outage. Its job is to report one.
The checklist
- ✅ Create paused, with no recipients. The object exists, it cannot fire, and nobody is wired to it.
- ✅ Verify by hand. Run the check, read what it actually asserts on.
- ✅ Backtest. Run the condition against history before it runs against now.
- ✅ Approve activation separately. Not the same change request that created it.
- ✅ Enable with the rollback ready. One object at a time.
- ✅ Reconcile. Nothing gets left paused by accident.
On paper the sequence is straightforward. The return paths are where most of the effort goes:
Create in the off position
Every monitor is created paused. Every detector is created disabled. And the recipient list is the last field you populate, not the first. Creation and activation are two different decisions, and combining them removes your only chance to inspect the thing before it can act. When a detector is created live, the first evidence you get about whether its threshold is sane is a page, and it is somebody else's page.
Most platforms support pausing directly. Where they do not, an alert created with no recipients at all is the fallback. That is weaker than a real disabled state, because a half-configured object in production is its own hazard, but it beats going live blind. The important part is not the API flag. It is that you have separated "the thing exists" from "the thing can wake someone up."
Verify before anyone sees it
Once the object exists in the off position, run it manually and read the result properly.
For a synthetic monitor that means executing the check against production and looking at the run in detail: did it complete, how long did it take, what did it actually assert on, and does the pass reflect the service being healthy or the assertion being too loose. A monitor that passes for the wrong reason is worse than one that fails honestly, because it will keep passing during the outage you built it for.
For a detector or alert rule, it means running the condition against history before it is allowed to run against the present. Some platforms give you this directly: as you configure the rule, the preview chart applies your condition to the stored signal and shows the windows where it would have entered the alerting state. Widen that preview to two or three weeks and count them. Where the platform has no preview, you can do the same thing by hand, by taking the rule's own query, bucketing it to the rule's evaluation interval, applying the same trigger condition, and counting what survives. The bucket has to match the interval. A rule that evaluates every five minutes, checked against hourly buckets, produces a number that means nothing.
Two things make the count readable. First, separate crossings from episodes: a condition that flips true, false, true across twenty minutes is one event and can produce eight notifications. Second, a count of zero is only informative if the window you checked contained an incident you can name. Over a quiet fortnight, zero tells you the fortnight was quiet.
Here's a manual backtest for an alert that fires when the HTTP 500 rate exceeds 5% in any five-minute window, run over the last three weeks:
The exact syntax varies by platform. The example below uses Splunk SPL because it makes the backtesting approach explicit, but the same idea applies regardless of your monitoring stack.
index=web_logs sourcetype=access_combined
| bin _time span=5m
| stats count(eval(status>=500)) as errors count as total by _time
| eval error_rate=if(total>0,round(errors*100/total,2),0)
| where error_rate>5
| stats count as breaching_windows
That counts breaching windows. To estimate distinct alert episodes instead, treat breaching windows separated by less than fifteen minutes as part of the same event:
index=web_logs sourcetype=access_combined
| bin _time span=5m
| stats count(eval(status>=500)) as errors count as total by _time
| eval error_rate=if(total>0,round(errors*100/total,2),0)
| where error_rate>5
| sort 0 _time
| streamstats current=f last(_time) as prev_time
| eval episode_start=if(isnull(prev_time) OR (_time-prev_time)>900,1,0)
| stats sum(episode_start) as episodes count as breaching_windows
The gap between those two numbers tells you which knob to turn. Sixty breaching windows across four episodes means the threshold is roughly right, but the rule probably needs a duration clause before it notifies. Sixty breaching windows across fifty episodes usually means the threshold itself is too sensitive.
Either way, this is the highest-value step in the sequence because it is the only one that tests the rule against reality rather than against your intent.
Activation needs its own approval
The change request that let you create the object should not be the change request that lets it fire.
At first glance this looks like process for its own sake. In practice it does something specific: it forces the person approving activation to look at evidence that did not exist when the work was first proposed. At creation time all anyone can review is your intent. At activation time there is a real object, a real manual run, and a real backtest. The second conversation is short, because you are handing over evidence rather than a plan.
It also gives you a clean place to stop. If the backtest looks wrong, nothing goes live and nothing needs reverting, because the object was never able to do anything.
Enable, one at a time
Enable one object and stay with it. Not the batch.
The first evaluation cycles after activation are where you find out whether the backtest matched reality, and you only get that signal if you are still watching. Enable twenty at once and the first firing arrives with no way to tell which change produced it, which is the position you spent the previous four steps avoiding.
Treat that first firing as information about the threshold rather than as a result. A detector that fires within minutes of going live is more often telling you something about your condition than about the service. Check the underlying signal before anyone acts on it. If the threshold turns out to be wrong, nothing has reached a human yet and the rollback line is already written.
GET before you write, always
Every write against a production monitoring API is preceded by a read of the current state of that same object, and the response is saved to a timestamped file.
This costs one extra call. Monitoring APIs are frequently full-replacement rather than partial-update, so a PUT built from a partially populated payload will quietly drop fields you never intended to touch: notification rules, tags, team assignments, custom properties. You will not see this in the response, because the write succeeds. You see it a week later when an alert routes to the wrong team.
mkdir -p backups
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
curl -sS -H "$AUTH_HEADER" "$API/detector/$ID" \
| tee "backups/detector-${ID}-${STAMP}.json" \
| jq -e '.id' > /dev/null
Two details in there matter. The object ID plus a UTC timestamp in the filename means that during a rollback you are not guessing which of four exports was the pre-change one. And the check on the end means an auth failure or an empty body cannot masquerade as a successful backup. A zero byte file in the backups directory is worse than no file, because you will trust it.
For bulk operations, export everything in scope first, as a single operation, before the first write goes out. Do not export as you go. If the script fails halfway through, the objects you have not reached yet are the ones you most need originals for.
Write the rollback first
The rollback for any monitoring change should fit on a single command, and it goes into the change record before the change is applied, with the real filename in it rather than a placeholder.
ID=abc123
BACKUP=backups/detector-${ID}-20260805T181200Z.json
curl -sS -X PUT "$API/detector/$ID" \
-H "$AUTH_HEADER" \
-H "Content-Type: application/json" \
--data @"$BACKUP"
Writing it afterwards defeats the point. The moment you need it is the moment you are least able to compose it, so the discipline is in having decided in advance what the undo is, not in the complexity of the command. For most of this work the undo genuinely is one line: disable the detector, pause the monitor, or PUT back the saved export.
If you cannot express the rollback in one line, that is useful information about the change. It usually means you are changing several objects at once and should be splitting the batch.
Scope the credentials
Use a token scoped to the specific environment and, where the platform allows it, to the specific capability. An org-wide admin token used for a routine dashboard update is a blast radius you are carrying around for no benefit.
This one is easy to skip because the wide token already exists and already works. The argument for the narrow one is not really about malice. It is about the script that has a bad loop bound, or the copy-pasted object ID from the wrong environment. A narrowly scoped token turns those into an error message instead of an incident.
The failure mode this creates
Every safety pattern introduces its own way of going wrong, and being honest about that is more useful than presenting the pattern as free.
Creating things in the off position means you can forget to turn them on. A monitor built during a migration, verified, approved, and then left paused because the activation window slipped, is invisible coverage debt. Nobody notices, because a paused monitor never complains. You find out during the incident it would have caught.
The fix is a reconciliation step, not a better memory. At the end of every change window, list every object in the paused or disabled state and account for each one: intentionally off, or dropped. Anything older than the window that nobody can explain gets either enabled or deleted. Most monitoring APIs will give you that list in one call, so this is a five minute task that closes the only real hole in the pattern.
I would rather carry that failure mode than the alternative one. Forgetting to enable a monitor is a gap you can find with a query. Enabling a bad detector is a gap that finds you.
What it actually costs
Yes, it is slower. Two approvals instead of one, an export step before every write, and a backtest that occasionally sends you back to redesign a threshold you thought was finished.
What you get in exchange is that the monitoring layer stops being a source of incidents. Across the migration and the detector rollout that followed, just over forty monitor definitions and twenty-six detectors went live across sixteen production applications. Nothing I enabled had to be rolled back for firing wrongly, and no change I made took out an existing alert path. That is not because the thresholds were inspired. It is because every one of them was inspected while it was still incapable of paging anyone.
Monitoring exists to reduce operational risk. When deploying it adds risk instead, the thing that needs fixing is the deployment process rather than the monitoring. The same pattern applies well beyond monitoring, and most infrastructure work has some version of the off position available. It is worth finding out what yours is before you need it.
If you run a variant of this, particularly the reconciliation step, I would like to hear how you handle it.
Top comments (0)