DEV Community: Slim

How to get alerted when someone changes your DNS records

Slim — Wed, 14 Oct 2026 14:35:51 +0000

TL;DR. Most DNS changes do not take your site down. A new MX record or an edited SPF record usually still resolves, so your uptime checks stay green. To catch these changes, list every record a name should have. Then get an alert when the live answer is different in any way. The original post has the same text with a short FAQ at the end.

A DNS change does not look like an outage

Uptime monitors wait for something to break. A DNS change often breaks nothing you can see.

Here is an example. Someone adds a second MX record to your domain. MX records tell other servers where to deliver your email. Maybe it was a test of a new email provider, and nobody removed it. Maybe someone stole the password to your DNS account.

Each MX record has a priority number, and senders try the lowest one first. If the new record has a lower number than yours, your email now goes to another server first. Your website works, your checks are green, and nobody notices for weeks.

Other changes work the same way:

Someone changes your nameservers at the registrar. Now a different server answers for your whole domain.
Someone edits your SPF record during a migration. SPF lists the servers that may send email as you, and your newsletter tool is no longer on it.
Someone deletes your _dmarc record while cleaning up old records.
Someone changes your CAA record. Now a different certificate authority can issue certificates for your domain.

These changes can leave your DNS lookups and website checks passing.

This happens in real attacks

In January 2019, the US Cybersecurity and Infrastructure Security Agency (CISA) published Emergency Directive 19-01. Attackers had changed DNS records on several US government domains.

They obtained credentials for a DNS account. They changed A, MX and NS records to point to their own servers. Then they could read web and email traffic as it passed through.

CISA's first order to the agencies was simple: check your public DNS records, and make sure each one points to the right place. Start with NS records.

A check like that tells you the records were right on one day. A monitor does the same check every few minutes, so you find out on the day of the change.

Why "does it resolve?" is not enough

Most DNS checks ask one of two questions.

The first is "Does the name resolve?" This catches a deleted record or an expired domain. It does not catch a changed record, because a changed record still resolves.

The second is "Does the answer contain this value?" For example, you enter your mail server, and the check fails if it disappears. This is better. But the check passes as long as your value is somewhere in the answer. So it misses:

an extra MX or NS record next to yours;
an SPF record changed around the part you check;
one A record replaced, when you have several.

To catch these, the check must know every record that should be there. Then it can fail on anything else.

Watch the exact record set

An exact record set is a list of every record that one name should have, for one record type. The check reads the live answer and compares it with your list:

A record in your list but not in the answer is missing.
A record in the answer but not in your list is unexpected.
A changed record shows up as both. The old value is missing and the new value is unexpected.

Any difference fails the check. The order of the records does not matter, and neither does the TTL.

To write the list, start from the records your domain has now. The DNS lookup tool shows them from Cloudflare and Google side by side. Check that every record is one you expect. If the two resolvers disagree, a change is still spreading, so wait until they match.

In Uptimepage, create a DNS monitor and choose exact set as the answer match. Write one record per line, the way a DNS answer shows it:

10 mail.example.com.
20 alt.example.com.

Say someone adds a third MX record. The next check fails, and the result shows what changed:

{
  "domain": "example.com",
  "record_type": "MX",
  "missing": [],
  "unexpected": ["5 mx.attacker.example."],
  "answers": ["10 mail.example.com.", "20 alt.example.com.", "5 mx.attacker.example."],
  "expected_contains": null,
  "matched": false
}

The new MX record has priority 5, lower than yours, so senders try it first.

The check ignores small differences that would only cause false alarms:

Upper and lower case in names do not matter, so 10 Mail.Example.com matches 10 mail.example.com.
The dot at the end of a name is optional.
Numbers and IP addresses are compared by value, so 2001:0db8::1 matches 2001:db8::1.

TXT values must match exactly, including case. In SPF and DKIM, every character counts.

Enter only the record value, without the name, TTL or type. For TXT records, you can paste quoted strings from your zone file, with all parts on one line.

Which records to watch

You do not need this for every record. Start with the ones where a quiet change hurts the most.

Record	Name	What it controls
NS	`example.com`	Which servers answer for your whole domain
MX	`example.com`	Where your email goes
TXT	`example.com`	Your SPF record: who may send email as you
TXT	`_dmarc.example.com`	Your DMARC policy: what receivers do with fake email from your domain
TXT	`selector._domainkey.example.com`	Your DKIM public key, which receivers use to check your email signatures
CAA	`example.com`	Which certificate authorities may issue certificates for you
A, AAAA	your server hostnames	Only if your servers have fixed IP addresses

Your main domain often has other TXT records too, such as verification records from other services. Put all of them in the list. A new one means someone connected a new service to your domain, and you want to know about that.

One monitor watches one name and one record type. Start with six monitors, one for each of the first six rows. Then add one for each other DKIM selector you use, and one for each fixed A or AAAA record set. A hostname with both an A and an AAAA record needs two monitors.

If you set up only one, make it NS. CISA put NS first too. Whoever changes your NS records controls all your other records.

Query your own nameserver

Normally a DNS lookup goes through a resolver, such as 1.1.1.1 or the one from your internet provider. The resolver keeps each answer for a while. This time is called the TTL. If your MX record has a TTL of one hour, the resolver can show the old answer for about an hour after a change. Sometimes it is longer: a resolver that cannot reach your nameserver may keep serving the old answer until it can.

You can skip this delay. Set the monitor's resolver to the IP address of your own nameserver, the one that holds your real records (your authoritative nameserver). Then every check sees your records as they are right now.

There is one exception: NS records. If someone changes your nameservers at the registrar, your own nameserver does not know. It still answers with the old, correct list. So watch NS through a public resolver, such as 1.1.1.1 or 8.8.8.8. A public resolver follows the registrar, like your visitors do. You can check what a public resolver sees for any name before you create the monitor.

When not to use an exact set

Some records change on purpose. An exact set on them only gives you false alarms.

Names behind a CDN or a load balancer get their IP addresses from a pool. The answer can change over time and depending on where you query from. I wrote about this in why DNS returns different IP addresses. An exact A set on such a name would fail all the time. Watch the CNAME record that points to the CDN instead.

SOA records change every time you edit your zone, because their serial number goes up. _acme-challenge TXT records come and go with each certificate renewal. Do not use an exact set for these either.

Keep the list next to your DNS

If you manage DNS with Terraform, put the monitor in the same repository:

resource "uptimepage_target" "mx" {
  name     = "mx example.com"
  interval = 300

  check = {
    type = "dns"
    dns = {
      domain           = "example.com"
      record_type      = "MX"
      expected_records = ["10 mail.example.com.", "20 alt.example.com."]
    }
  }
}

Now a pull request that changes an MX record must change the monitor too. An MX change that nobody reviewed fails the check. You need Terraform provider 0.15.0 or later. Read more in monitoring as code. All DNS options are in the DNS monitor docs.

Where Uptimepage fits

I build Uptimepage, an uptime monitor that checks your services from several regions and alerts you when one fails. The exact set match above is part of its DNS monitor. A DNS change alerts you once the failure is confirmed: by default two failed checks in a row, from more than half of the regions. The result names the missing and the unexpected records, so you see the change without running dig yourself. It is open source under AGPL, so it can also run on your own server.

Whatever you use, look up your NS records today and check that every name in the answer is one you expect.

Now the question for the comments: has a DNS record on one of your domains ever changed without anyone noticing? I would like to hear how you found out, and how long it took.

Google says on-call needs eight engineers. You have three.

Slim — Tue, 06 Oct 2026 14:21:20 +0000

TL;DR. Google's SRE book works out that a 24/7 on-call rotation needs at least eight engineers. A team of two or three cannot reach that number, so it has to change what the number is built from: which alerts page a person, which hours are covered, and how fast someone must answer. After that it needs one named person on call at any moment and a short escalation ladder behind them for the page nobody answers. A weekly count of pages shows whether it is working. The original post has the same text with a short FAQ at the end.

"We are not Google"

Google's first SRE book has a chapter called Being On-Call. Readers wrote back. The follow-up book, the SRE Workbook, opens its own on-call chapter with their feedback, and the first item on the list is this:

"We are not Google; we're much smaller. We don't have as many people in the rotation, and we don't have sites in different time zones. What you described in your first book is irrelevant to me."

I build Uptimepage, which pages people when a monitor goes down, so I had to decide what a rotation should look like for a team that small. That reader is right about the numbers. The reasoning behind the numbers still works for three people, and it leads to a different setup.

Where the eight comes from

The book starts from a limit on time:

"we strive to invest at least 50% of SRE time into engineering: of the remainder, no more than 25% can be spent on-call"

Then it does the arithmetic:

"Assuming that there are always two people on-call (primary and secondary, with different duties), the minimum number of engineers needed for on-call duty from a single-site team is eight: assuming week-long shifts, each engineer is on-call (primary or secondary) for one week every month."

Two people on call, each for at most a quarter of the time, makes eight. Run the same arithmetic on a small team, with one person on call at a time and week-long shifts:

Team size	Share of weeks each person is on call
2	50%
3	33%
4	25%
8, with two on call at a time	25%

A team of four reaches Google's limit only by giving up the secondary. A team of three is over the limit before anything breaks, and a team of two is on call every other week.

A schedule cannot fix that, because any rotation of three people still has three people in it. What the team can change is how often the pager goes off, during which hours, and how quickly it has to be answered.

The limit exists because pages are expensive. The same chapter estimates that one incident, with its diagnosis, fix and follow-up, takes about six hours of work, which is why it allows at most two incidents in a 12-hour shift. A person on call every third week can live with that week if it is quiet. So the work for a small team is to make the week quiet, then make sure the rare page reaches someone.

What a team of three can change

Decide what is worth a page

The Workbook is direct about this:

"Engineers shouldn't have to be at a computer and working on a problem within minutes of receiving a page unless there is a very good reason to do so."

Its Table 8-1 gives three examples of a realistic response:

Incident	Response time
Revenue-impacting network outage	5 minutes
Customer order batch processing system stuck	30 minutes
Backups of a database for a pre-launch service are failing	Ticket (response during work hours)

Sort every alert you have into those three rows. Only the first two should reach a phone at night. The rest go to a channel or an inbox that somebody reads in the morning.

An alert has to pass two more filters before it may page. It has to be real: one probe with a bad network path should not wake anyone. And it has to come with something to do. The SRE book's wording is "All paging alerts should also be actionable." A page that can only be watched until it clears teaches people to ignore pages.

Name one person at a time

The usual start for a small team is a shared channel: every alert goes to #alerts, and everyone is in it. During the day this works, because somebody is looking. At 03:00 five people receive the message, and each of them can reasonably think one of the other four has it.

A rotation puts exactly one name on call at any moment. That person knows it is them, and everyone else knows it is not. The channel can still get a copy, but the page goes to the person.

Hand off on a weekday morning, when both people are awake and can talk about what is still open. A handoff at midnight on Sunday means nobody mentions the disk that is filling up. Keep it at the same local time all year too, so the change does not move by an hour when the clocks do.

Split the day before you split the week

The Workbook does not expect one person to carry a full week around the clock:

"In our experience, 24 hours of on-call duty without reprieve isn't a sustainable setup."

Its suggestion fits a small team too: "it would be better for two engineers to split a week of on-call, with one person on-call during the day and one on-call overnight."

With three people that can be two rotations in one schedule. Working hours rotate weekly among all three. Nights and weekends rotate separately, with the order shifted by one, so nobody carries both in the same week.

There is also a cheaper option: leave some hours uncovered on purpose. If your customers are businesses in one region and nothing you run loses data while it is down, a page at 02:00 may only move the fix from 08:00 to 02:30. Decide that as a team, then write it where customers can see it, in your SLA and on your status page. As a customer I would rather read a stated response window during European working hours than a 24/7 promise kept by one tired person.

Do the arithmetic before you choose. A 99.9% target leaves 43 minutes a month. One night outage that waits six hours for the morning uses more than eight months of that budget. If the number you promised cannot absorb that, someone has to be on call at night.

Add a ladder for the unanswered page

One name on call is one point of failure. Phones run out of battery, and a person who was paged twice last night can sleep through the third.

An escalation policy is an ordered list of who gets paged next when a page goes unanswered. Level one pages when the incident opens, and each level waits a set time for an acknowledgement before the next one pages. Acknowledging stops the ladder.

For three people a ladder can be this short:

The person on call. Wait 10 minutes.
The other two. Wait 15 minutes.
Everyone, on the loudest channel each person has.

How long to wait comes from the response time you promised. The SRE book's typical values are "5 minutes for user-facing or otherwise highly time-critical services, and 30 minutes for less time-sensitive systems". If level one has 5 minutes to answer, the wait before level two cannot be 30.

This ladder is the small team's version of Google's secondary. Nobody is formally second on call, and the page still has somewhere to go. Watch how often level two fires. If it fires most weeks, the other two people are on call as well, whatever the schedule says.

Make swaps cheap

The Workbook again:

"No one can promise on Monday that they won't have the flu on Thursday."

On a team of eight a swap is easy to arrange. On a team of three every swap lands on one of two people, so it has to take a minute and leave no doubt about who holds the pager afterwards. That means an override on the schedule itself, for exact dates, which ends without anyone remembering to undo it. A chat message asking a colleague to cover Thursday changes nothing about where the page goes.

Put the shifts in everyone's calendar as well. A person who sees an on-call week next to a dentist appointment three weeks ahead asks for the swap now instead of on the day.

Test that the page arrives

A schedule can name a person the page cannot reach. Their phone is on do-not-disturb, the chat app is muted at night, or the email address was never confirmed. Everything looks covered until the first night page arrives on a silent phone.

So test it. On the first morning of a shift, send a test page to the person on call and have them acknowledge it from the phone they will have at night, with the settings they will have at night. Use at least one channel that is allowed to make noise through do-not-disturb, such as SMS from a number saved as an emergency contact, or a push app with an emergency priority.

GitLab's 2017 outage shows what an untested path costs. The failure alerts for its backup job were sent every night and rejected by the receiving mail server, and nobody knew for months.

Count the pages

Google's teams work to a number: "We target a maximum of two incidents per on-call shift". The Workbook also describes a team with a budget of two paging incidents per shift that had been receiving five for a year. Their answer was a project to reduce the pages, approved by senior management.

A small team needs that count more than a large one does, because each page lands on a third of the team. Once a week, look at how many pages there were, how many of them needed a person, and how long the first acknowledgement took. A page that needed nobody is an alert to fix or delete. A slow acknowledgement is a reason to look at the ladder and the paging channel before it is a reason to look at the person.

What this does not fix

Two people are still on call every other week, and no ladder changes that. What is left is to cover fewer hours or fewer systems and tell customers which, and to pay for the nights. The Workbook's position is short: "On-call usually implies some amount of out-of-hours work. We believe this work should be compensated."

Four is the first team size where one person on call at a time meets the 25% line. If on-call is wearing down a team of three, that is a real argument for the fourth hire, and the table above is how to make it.

Sources

Google, Site Reliability Engineering, chapter 11, Being On-Call.
Google, The Site Reliability Workbook, chapter 8, On-Call.
GitLab, Postmortem of database outage of January 31, 10 February 2017.

Where Uptimepage fits

I build Uptimepage. It checks your services from several regions, and when one goes down it pages whoever is on call.

An on-call schedule holds one or more rotations, each on call at all hours or only in the hours you give it. Daily and weekly handoffs stay at the same local time across daylight saving. Overrides cover exact dates. Each person picks the channels that page them, and the schedule list names anyone on a rotation whom no channel can reach. An escalation policy is a ladder of levels with a wait on each, and acknowledging from the chat or email alert stops it. Each person gets a calendar link for their shifts, and the incident reports include mean time to acknowledge.

One gap to know about: there is no phone-call channel yet. SMS, Pushover's emergency priority, Telegram and the chat apps are there, and if you already use PagerDuty you can put it on a level. The incident docs have the details. It is open source under AGPL, so it can also run on your own server.

Whatever you use, write down the answer to one question. At 03:00 tonight, whose phone rings, and whose rings next if they do not answer?

Now the question for the comments: how does a team of two or three cover nights where you work? I would like to hear from teams that leave some hours uncovered on purpose and say so on their status page.

Uncle Bob's uml-viewer on Rust: 35 dependency cycles down to 3

Slim — Tue, 22 Sep 2026 06:58:19 +0000

TL;DR. Uncle Bob released uml-viewer, a clickable architecture diagram that draws Dependency Rule violations in red and is meant to be driven together with a coding agent. It only parses Clojure, so I wrote a 200-line script that turns my Rust crate into its input file. The first picture showed 167 red arrows and 35 dependency cycles. Six commits and 315 files later: 109 red arrows, 3 cycles, and app.rs down from 1,087 lines to 642. No behavior changed. The tool matters less than the loop it forces on you: look, point at one arrow, let the agent move code, look again. The original post has the same text with a collapsible FAQ and the source list at the end.

Uncle Bob's argument for the tool is short: agents still need supervision, and reading every line they write is the bottleneck. So supervise the structure instead of the text: draw the system, find the shape that is wrong, tell the agent to fix that shape, and check the new drawing.

I wanted to know if that works on a real codebase, so I pointed it at Uptimepage, about 146,000 lines of Rust in one crate. It does, with one condition: the picture is only as honest as the input file you feed it.

Five of the 167 red arrows and two of the 35 cycles, and where the code went. Every arrow that pointed up now points down.

What uml-viewer is

uml-viewer is an open-source desktop tool by Uncle Bob (Robert Martin) that draws a codebase as a UML-like diagram you can click. It is written in Clojure and needs the Clojure CLI and Java 21 or newer.

Namespaces are components. The modules inside a namespace are the component's elements. Nesting can go as deep as your source tree does. You double-click a component to open the next level, double-click a module to see its class card, and click a function to open the source file at that line.

Color comes from CRAP and mutation scores, so a red box is code with high complexity and weak tests, or code with no metrics at all, which the README counts as the worst grade. Red arrows come from the Dependency Rule: you tell the tool which namespaces sit at which architectural level, and every dependency that points from an inner level to an outer level is drawn red.

The tool is built to run next to an agent. By default it opens a tmux window with Grok in the examined project and the two talk through a small mailbox directory. You can also drive it by hand from any agent session: regenerate the input file, press R in the viewer, and the diagram reloads.

Getting a Rust crate into it

The only parser it ships reads Clojure. That sounded like the end of the experiment, until I read what the viewer actually consumes. It never sees source code. It reads one EDN file: a list of classes with a namespace and a level, a list of edges with a from, a to and a kind, and a list of levels. That is a format any script can emit.

So I asked the agent to write a generator for Rust. It came out at about 200 lines of Python and does four things:

Every .rs file is one class. mod.rs stands for its directory and lib.rs is skipped.
Every use crate::…, super::… and self::… path, plus every inline crate::a::b path in a function body, becomes a dependency edge to the nearest enclosing module that exists as a file.
Comments are stripped first, so a path mentioned in a doc comment does not count as a dependency.
A hand-written LEVELS list gives each top-level module a rank.

The last item is the only opinion in the script:

# inner (high level) first, matching the Dependency Rule ranks
LEVELS = [
    ["domain", "error", "text", "metric_names"],
    ["storage", "security", "net", "http_client", "config",
     "quotas", "observability", "pagination"],
    ["pipeline", "escalation", "notifier", "email", "telegram",
     "whatsapp", "billing", "auth", "analytics",
     "jobs", "http_outbound", "targets"],
    ["api", "web", "mcp", "marketing", "agent", "worker", "scheduler",
     "request", "public_status", "templates", "oauth",
     "ad_hoc_dispatch", "channels"],
    ["app", "router", "bootstrap", "main", "bin"],
]

Rank 0 is vocabulary that everyone may use: domain types, error codes, text helpers. Rank 1 is infrastructure: storage, security primitives, config. Rank 2 is the services that do the work: probing, escalation, notification, billing. Rank 3 is every way into the system: HTTP handlers, HTML views, the MCP server, the marketing site. Rank 4 is assembly: the app state, the router, main.

The rule is then mechanical. An edge is a violation when the module it comes from has a smaller rank than the module it points to. Same rank is allowed. Foreign crates like axum and sqlx are ovals outside the diagram and are never compared.

The picture is only as honest as the levels list

The generator does not decide the architecture. I do, in that list. If you put api in the same group as domain, the diagram turns green and you have learned nothing. Write the list you want to be true, then let the red arrows show how far the code is from it.

What the first picture showed

The first diagram had 167 red arrows, 35 pairs of modules that imported each other, 17 modules importing api, and an app.rs of 1,087 lines.

Behind the numbers were shapes I half knew about and had never seen drawn:

storage reached up into api for error codes and dashboard read models, and into worker for heartbeat state.
api and web imported each other. The JSON side took session and token extractors, cookies and client IP from the HTML side, and the HTML side took the heartbeat read model from the JSON handlers.
quotas read accounts and organizations from storage, while storage embedded quota SQL fragments and read plan types from quotas. A cycle in both directions.
marketing and oauth imported web for three template filters.
config imported auth and billing for two enums.

None of this was visible in the review of any single pull request. Every one of those imports was reasonable on the day it was written. A diff shows one import at a time, so the sum of them never appeared in any review.

The loop

Every round had the same six steps:

Regenerate the EDN file and reload the viewer.
Pick one red arrow. Hover it to see which module pairs it bundles.
Tell the agent one thing: what must not import what, and where the shared piece should live.
The agent moves the code and runs the compiler and the tests.
Regenerate. Check that the arrow is gone and count the new ones.
Run the full suite, review the diff, commit.

The instruction in step 3 is the part that makes this work. It looks like this:

storage must not import api. The error codes and the dashboard read models
it takes from there are crate-wide vocabulary. Move the codes to error::codes
and the read models to domain::metrics, and point every reader at the new
path. No behavior changes.

That is one rule, one violation, and one place to put the result. The agent does not have to guess what "cleaner" means. It has a named arrow to remove, and the next diagram says whether it did.

I ran this with Claude Code, but nothing in the loop depends on it. Codex, Cursor, Grok, or any agent that can edit files and run a test suite gets the same instruction and the same picture afterwards.

Six rounds took the crate from 167 red arrows to 109:

Round	Red arrows I pointed at	Where the code went
1	`storage` into `api` and `worker`, `config` into `auth`	error codes to `error::codes`, 12 read models to `domain::metrics`, provider enums to `domain::credential`
2	`storage` into `auth` and `api`	token hashing, HMAC and SHA-256 helpers to `security`, redaction scrubbers to `security::redaction`
3	`api` and `web` into each other	extractors, cookies, client IP and host resolution to a new `request` module that imports neither
4	`storage` and `quotas` into each other	SQL fragments to `storage::count_sql`, plan types to `domain::quota`
5	`marketing`, `oauth` and `api` into `web`	template filters and formatters to `templates`, `app.rs` split into `config::boot`, `observability::readiness`, `targets::status` and `net`
6	`public_status` and `web` into `api`, `security` into `http_client`	page envelopes to `pagination`, `Cipher::from_config` into `security`

And the totals:

Six commits, 315 files, same test suite before and after. Violations down 35 percent, cycles down 91 percent.

	Before	After
Dependency Rule violations	167	109
Module pairs importing each other	35	3
Modules importing `api`	17	4
Lines in `app.rs`	1,087	642
Files touched		315
Behavior changes		0

I regenerated the graph at every one of the six commits to see what each round removed:

Cycles and api importers per round. The first two rounds did most of the work on the red count; rounds three to six were about the cycles.

The shape surprised me. The red arrow count stopped moving after round two. Rounds three to six barely touched it, but they took the cycles from 23 down to 3, because a cycle between two modules at the same level is not a Dependency Rule violation at all. The rule catches arrows that point up and says nothing about two modules on the same level that import each other, so you need both counts.

Not every move survived. In a later round the agent moved the health-check paths into observability, and a coupling test on the marketing module said no, because that module is only allowed to reach a short list of leaf modules. Another move put the strict JSON parser under request, then had to come back because that parser reads the OpenAPI document. Both reverts took minutes, because the diagram and the test said so before the commit did.

Why I stopped at 91

After the six commits, 109 red arrows were left. Most pointed into app, but ten did not: a sampler and a silence job that lived under observability but reached into the scheduler and the notifier, two background jobs reading public_status, the HTTP metrics layer, and the rate-limit middleware in quotas reading the app state. That middleware was also the third of the three cycles. One more round moved each of those into the module that owns it and left 91 red arrows and 2 cycles.

Every one of the 91 is the same shape. Handlers read the app state, and the app state is built from the modules those handlers live in. app imports api and web to mount them; api and web import app to get the state.

Fixing it is possible. Split the state into per-handler sub-states and hand each handler only its slice. I counted what that would touch: about 90 files of plumbing that make the code harder to read, not easier. I stopped because I could name every remaining arrow, and at that point the diagram had told me everything it was going to.

Stop when every arrow has a name

Drive the count down until every arrow that is left has a name and a reason, then stop. The 91 arrows left in my crate are all "this handler reads the app state", and a diagram that shows them is more honest than one that hides them behind 90 files of indirection.

Keeping it fixed

Two things stop the graph from drifting back.

The marketing module has a test that is an allow-list: the exact set of leaf modules it may import. A new reach into the app fails the build. It used to be a deny-list, which only catches the mistakes you already thought of.

The other is a habit, not a test. Regressions arrive with feature commits, not with refactors. After every push I regenerate the graph and read the list of red arrows that do not point into app. If the list is empty, the feature stayed in its layer. If it is not, the offending edge is one instruction away from gone, before the next feature builds on it.

Why this beats "refactor the codebase"

Two months ago I wrote about mapping this codebase for humans and AI agents. The lesson then was that a model is good at shape and bad at numbers, so I had to check every count it produced by hand.

This is the same lesson, applied to refactoring. Here the agent never counts anything; the generator does. The agent gets one rule with one violation and a picture that says pass or fail after every change. It is the same reason a failing test is a better instruction than a paragraph of requirements: the acceptance criterion exists before the work starts, and it is not the agent that judges it.

What I did not get is the color. CRAP and mutation scores come from Uncle Bob's Clojure tooling, and there is no Rust equivalent wired in yet. The README is clear that a box with no metrics is painted as the worst grade, so the fill on my boxes means nothing until coverage and mutation numbers exist for Rust. The arrows alone were worth the setup.

Do it yourself

Install the Clojure CLI and Java 21 or newer, then clone uml-viewer.
Write a generator for your language. One class per module, one dependency edge per import, output as EDN with :hierarchical true. The generated examples/uml-viewer.edn in the repo is a complete example of the format. In Rust, imports are use paths; in TypeScript they are import statements; in Python, import and from. A regex over each file is enough to start.
Write the :levels list by hand, inner layer first. Describe the architecture you want; the red arrows will show where the code differs.
Run the viewer against your file. On a fresh start it waits for its companion agent; the --restart flag restores the last view without spawning one, which is what you want when you drive it from your own agent session.
Pick one red arrow. Write one instruction that names the two modules and the new home. Regenerate. Repeat.

Key takeaways

uml-viewer reads a plain EDN file, not source code. A 200-line script gets any language in.

The levels list is the only opinion in the input, so write the architecture you want and let the red show the distance.

Give the agent one rule, one violation and one destination per instruction. The next diagram is the acceptance test.

Rust will not tell you two modules import each other. Build the graph and count the mutual pairs yourself.

Stop when every remaining red arrow has a name. Mine are all "handler reads app state", and that is fine.

Turn the result into an allow-list test, and regenerate the graph after every feature push, because that is when regressions arrive.

The six commits are public in the Uptimepage repository if you want to read what moved and why.

Sources

Robert C. Martin, uml-viewer on GitHub. The README documents the EDN format, the :levels rule and the companion agent mailbox.
Robert C. Martin, The Clean Architecture, The Clean Code Blog, August 2012. The Dependency Rule.
Uptimepage, source repository, AGPL. Commits bda59342 through cee12c44 are the six rounds described here.

I build Uptimepage because I wanted uptime checks that say why something failed instead of "transport error": HTTP with per-phase timing, TLS and domain expiry, ping and TCP, heartbeats for background jobs, and browser flows, from several regions. It is AGPL-3.0 open source, which is why the six commits above are public and you can check every number in this post against them.

Now the question for the comments: what draws the module graph for your language, and does it show the cycles or only the layer violations? My rounds three to six only moved the cycle count, and I would have missed that with a tool that colors arrows but never counts pairs.

ICMP vs TCP vs UDP: the difference, explained for developers

Slim — Sun, 13 Sep 2026 09:49:11 +0000

Originally published on the Uptimepage blog. I build an uptime monitor in Rust, and its ping, TCP and DNS checks are these three protocols with a timer on them, so I had to get the differences straight.

TL;DR. ICMP, TCP and UDP all travel inside IP packets, but they ask different questions. ICMP echo (ping) asks "does this address answer at all?" and needs no port. TCP asks "is a program listening on this port?" and gets a clear yes, a clear no, or silence. UDP asks nothing by itself: you only learn something if the program on the other side chooses to reply. A closed port answers differently on each one, and that difference is most of what a network check can and cannot tell you. The original post has the same text with a collapsible FAQ and the RFC list at the end.

One address, three ways to knock

Every packet on the internet is an IP packet. IP knows how to move a packet from one address to another and nothing else. One byte in the IP header says what is inside, and IANA keeps the list: ICMP is protocol 1, TCP is protocol 6, UDP is protocol 17. The three are siblings: same addresses, same routers, and a different promise about what happens once the packet arrives.

Three ways to knock on one address. The last row of each lane is what silence means there.

	ICMP	TCP	UDP
Specification	RFC 792 (1981), RFC 4443 for IPv6	RFC 9293 (2022)	RFC 768 (1980)
IP protocol number	1 (58 for ICMPv6)	6	17
Ports	none	source and destination	source and destination
Handshake	none	three packets before any data	none
Delivery guarantee	none	ordered, complete, retransmitted	none
Header	8 bytes for an echo	20 bytes minimum	8 bytes
Typical users	ping, traceroute, error reports	HTTP/1.1 and HTTP/2, TLS, SSH, SQL, SMTP	DNS, NTP, QUIC and HTTP/3, video, games

ICMP: the network talking about itself

ICMP is not a transport. RFC 792 says it plainly: ICMP "is actually an integral part of IP, and must be implemented by every IP module." It exists so that routers and hosts can report on delivery: this destination is unreachable, that packet ran out of hops. ICMP messages have a type and a code but no ports, because nothing sits on top of ICMP waiting for them. The IP stack consumes them itself.

The one ICMP message everybody has typed is the echo. ping sends an echo request (type 8, or 128 in ICMPv6) and the far host's kernel sends back an echo reply (type 0, or 129). No program had to be running. RFC 1122, the host requirements standard, makes it a rule: "Every host MUST implement an ICMP Echo server function that receives Echo Requests and sends corresponding Echo Replies."

$ ping -c 1 uptimepage.dev
PING uptimepage.dev (204.168.246.94): 56 data bytes
64 bytes from 204.168.246.94: icmp_seq=0 ttl=50 time=96.069 ms

A reply proves two things: the address belongs to a machine that is switched on, and packets can travel there and back. It also gives you the round-trip time for free.

That is all a ping proves. It says nothing about whether a web server, a database or anything else is running, and a host can answer every ping while every service on it is dead.

Two things about ICMP surprise people. The first is that no reply does not mean down. Firewalls drop ICMP all the time. On AWS a fresh security group blocks it, and the documentation says so: "To ping your instance, you must add one of the following inbound ICMP rules." Before you rely on ping for a host, confirm the host answers a ping at all.

The second is that sending ICMP needs privilege. A raw socket needs CAP_NET_RAW. Linux has an unprivileged alternative, the ICMP datagram socket, but the kernel documentation says the default ping_group_range is "1 0", "meaning, that nobody (not even root) may create ping sockets." Docker opens that range in every container that gets its own network namespace, which is why ping works there without anyone thinking about it; a container on --network host inherits the host's setting instead. I learned the limits of that the slow way: the same probe binary that pinged fine in Docker could not open an ICMP socket at all on a Firecracker microVM, until I gave it CAP_NET_RAW.

TCP: a promise before the first byte

TCP is a connection. Before either side sends a byte of data, the two ends run the three-way handshake: SYN, SYN-ACK, ACK. RFC 9293, which replaced the 1981 specification in 2022, calls it "the procedure used to establish a connection." It costs one round trip, and that round trip pays for every promise TCP makes afterwards: bytes arrive in order, lost segments are sent again, the receiver is not flooded (flow control), and the network is not flooded (congestion control). To the program it looks like a stream. You write bytes on one side and read the same bytes, in the same order, on the other.

This is what HTTP/1.1, HTTP/2, TLS, SSH, PostgreSQL, MySQL, SMTP and most of what you deploy run on.

The handshake is also a good probe, because a SYN gets one of three answers:

SYN-ACK. A program is listening. The connect succeeds.
RST. Nothing is listening. RFC 9293 says that if the connection does not exist, "a reset is sent in response to any incoming segment except another reset." Your connect fails at once with "connection refused".
Nothing. A firewall dropped the SYN, or the host is gone. The kernel keeps retrying. On Linux the default tcp_syn_retries is 6, and the kernel documentation puts the final timeout for an active connection attempt at 131 seconds.

Here are all three from my laptop. The last one has a three second limit; without it macOS waits 75 seconds.

$ nc -zv uptimepage.dev 443
Connection to uptimepage.dev port 443 [tcp/https] succeeded!

$ nc -zv 127.0.0.1 9
nc: connectx to 127.0.0.1 port 9 (tcp) failed: Connection refused

$ nc -zv -G 3 uptimepage.dev 8443
nc: connectx to uptimepage.dev port 8443 (tcp) failed: Operation timed out

Nmap's documentation gives these states the names everyone uses: open, closed and filtered.

What a successful handshake proves: a process accepted a connection on that port. Not that it is healthy. A database that accepts connections and then rejects every query still passes. That is why a TCP check belongs on things that have no HTTP surface (SSH, SMTP, a database port) and an HTTP check belongs on things that do.

UDP: a message, nothing more

UDP is the smallest transport there is. RFC 768 fits on three pages. It adds a source port, a destination port, a length and a checksum to an IP packet, 8 bytes in total, and stops. The RFC says: "The protocol is transaction oriented, and delivery and duplicate protection are not guaranteed." There is no handshake, no acknowledgement, no ordering and no retransmission, and no congestion control either. RFC 8085 opens by saying UDP "has no inherent congestion control mechanisms" and then spends more than fifty pages telling application designers how to add their own.

Why anyone uses it: latency and control. A DNS lookup is one packet out and one packet back, no handshake. RFC 1035 capped a DNS answer over UDP at 512 bytes. EDNS(0) (RFC 6891) later let a client advertise a bigger buffer, so most answers fit in one datagram today; one that still does not fit is cut short with the TC bit set, and the client asks again over TCP (RFC 7766). NTP, most game traffic and real-time video use UDP because a late packet is worth less than the next one, so waiting for a retransmission is the wrong trade. And QUIC, the transport under HTTP/3, is built entirely on UDP. RFC 9000 states that "QUIC packets are carried in UDP datagrams" and gives the reason as "to better facilitate deployment in existing systems and networks". In practice that means QUIC does its own handshake, loss recovery and congestion control in the application, and can change them without waiting for every operating system to update its TCP stack.

$ dig @1.1.1.1 uptimepage.dev A +noall +answer +stats
; <<>> DiG 9.10.6 <<>> @1.1.1.1 uptimepage.dev A +noall +answer +stats
; (1 server found)
;; global options: +cmd
uptimepage.dev.     3600    IN  A   204.168.246.94
;; Query time: 125 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Wed Sep 02 11:40:44 EEST 2026
;; MSG SIZE  rcvd: 59

One datagram out, one datagram back carrying a 59-byte answer (67 bytes with the UDP header), and no handshake before it. Over TCP the same lookup costs two round trips, one for the handshake and one for the query, which is why DNS defaults to UDP.

Now the probing problem. Send a UDP datagram to a port and what comes back?

If nothing listens there, the host's IP stack should answer with an ICMP error. RFC 1122 again: "If a datagram arrives addressed to a UDP port for which there is no pending LISTEN call, UDP SHOULD send an ICMP Port Unreachable message." So "closed" is detectable, and it is ICMP that detects it.

If something listens, you get whatever that program chooses to send. A DNS server sends an answer. A syslog receiver sends nothing, ever. So silence means open, or filtered, or dead, and Nmap has a state for this, open|filtered. Its documentation explains: "This occurs for scan types in which open ports give no response."

The consequence is that a generic "UDP port check" does not exist. To check a UDP service you have to speak its protocol: send a real DNS query and read the answer, send a real NTP request and check the timestamp. This is why monitoring tools have a DNS check and a TCP check but rarely a UDP check.

The same closed door, three answers

Situation	ICMP echo	TCP SYN	UDP datagram
Service up	echo reply	SYN-ACK	the program's reply, or nothing
Port closed, host up	echo reply (ICMP has no ports)	RST, "connection refused"	ICMP port unreachable
Firewall drops the packet	silence	silence, retried for about two minutes	silence
Host gone	silence, or an unreachable error from the last router	silence, or that same ICMP error surfacing as "no route to host"	silence, or that same ICMP error

Read the table by columns. TCP is the only one where "closed" and "filtered" look different on their own, UDP borrows ICMP to say "closed", and ICMP cannot see ports at all. Silence is the ambiguous answer on all three, which is why no check should conclude "down" from one silent probe sent from one place. Two or three consecutive misses, seen from more than one network, is the honest bar. I wrote about that in how to stop false uptime alerts.

What this means for a monitor

I build Uptimepage, and its check types map onto these questions, so this is how the theory turns into a config.

Ping sends one ICMP echo with a 32-byte payload, the size Windows ping uses (Unix ping sends 56, as the transcript above shows), so middleboxes see ordinary diagnostic traffic. It tries the first IPv4 and the first IPv6 address of the host and splits the timeout between them. Silence for the whole budget is down, because an echo has no way to refuse. Use it for routers, gateways and hosts that expose no service.
TCP runs the handshake and closes the connection. Accepted is up, refused is down, and a timeout is recorded as an error with the time it took. An error counts as a failed check for alerting exactly like down, so a firewalled port still opens an incident once the confirmation threshold is met. Use it for databases, brokers, SSH and mail, where "a process accepted the connection" is the most you can assert from outside.
DNS sends a real query and checks the answer's content, so it is the UDP check for the one UDP service almost everyone depends on. It can point at a specific resolver, which matters when you want that server's view and not your cache's.
HTTP is a TCP handshake, then TLS, then a request, and it records the time spent in each phase: DNS, connect, TLS and time to first byte. The connect phase is the handshake from this article, measured inside a bigger check.

Pick the check that asks the question you care about, and remember what each one cannot see. There is a fuller version of that decision in the monitor types documentation, and a story-shaped version in the mystery of the "down" website.

Common questions

Is ICMP a transport protocol like TCP and UDP?

No. ICMP is a control protocol that belongs to IP itself, and RFC 792 says every IP module must implement it. It has no ports and carries no application data. It reports on delivery: a destination is unreachable, a packet ran out of hops, and the echo pair that ping uses.

Why does ping work while my website is down?

Because ping only proves that the host's IP stack answers and that packets can travel there and back. The kernel sends the echo reply, so no web server, database or other program has to be running. A host can answer every ping while every service on it is dead.

Why does a closed TCP port fail at once but a firewalled one hangs?

A closed port sends back a TCP reset, so the connect fails immediately with connection refused. A firewall that drops the SYN sends nothing, and the kernel keeps retrying. On Linux the default is six retries, and the kernel documentation puts the final timeout at 131 seconds.

Can I check a UDP port the way I check a TCP port?

No, because an open UDP port sends nothing back unless the program behind it chooses to answer. A closed port is detectable, since the host should reply with an ICMP port unreachable message, but open and filtered both look like silence. To check a UDP service you have to speak its protocol, for example send a real DNS query.

If UDP is unreliable, why does HTTP/3 use it?

UDP itself gives no delivery guarantee, and QUIC adds its own on top. QUIC runs inside UDP datagrams and does its own handshake, retransmission and congestion control in the application, which lets it evolve without waiting for every operating system to update its TCP stack.

Sources

IETF, RFC 792: Internet Control Message Protocol, September 1981.
IETF, RFC 4443: ICMPv6 for the Internet Protocol Version 6, March 2006.
IETF, RFC 768: User Datagram Protocol, August 1980.
IETF, RFC 9293: Transmission Control Protocol, August 2022.
IETF, RFC 1122: Requirements for Internet Hosts, Communication Layers, October 1989. Sections 3.2.2.6 and 4.1.3.1.
IETF, RFC 8085: UDP Usage Guidelines, March 2017.
IETF, RFC 1035: Domain Names, Implementation and Specification, November 1987. Section 4.2.1.
IETF, RFC 6891: Extension Mechanisms for DNS (EDNS(0)), April 2013.
IETF, RFC 7766: DNS Transport over TCP, Implementation Requirements, March 2016.
IETF, RFC 9000: QUIC, a UDP-Based Multiplexed and Secure Transport, May 2021.
IANA, Assigned Internet Protocol Numbers.
Linux kernel, IP sysctl documentation, tcp_syn_retries and ping_group_range.
Nmap, Port scanning basics, the six port states.
Amazon Web Services, Security group rules for different use cases, rules for ping/ICMP.

I build Uptimepage because I wanted these checks for my own services: ping, TCP, DNS and HTTP with per-phase timing, plus TLS and domain expiry, heartbeats for background jobs, and browser flows, from several regions. It is AGPL-3.0 open source, so you can run it on your own server if you prefer.

Now the question for the comments: do you still run ping checks on anything, and what for? I keep them for routers, gateways and hosts that expose no service, and almost nothing else. If a ping check has earned its keep for you somewhere I have not listed, I would like to hear where.

Your domain can expire while your uptime monitor stays green

Slim — Tue, 04 Aug 2026 14:15:27 +0000

TL;DR. One HTTP check on your homepage is not enough to know your site works. It cannot tell you when your domain expires, because an expired domain still returns 200 OK from a parking page. Mail dies at the same time, just as quietly, and you keep losing money while every monitor stays green. The fix is not one clever check. It is a few plain ones, starting with the two that fail on a known date: domain and TLS expiry.

Two checks green, the server behind them is down. A status check cannot tell the difference.

I keep seeing new products treat reliability as something that takes care of itself. You ship the site, you watch it load once, and you assume it will keep loading. Hosted once, up forever. That is the most expensive assumption a young team can make, because nobody is watching on the day it stops being true.

So ask one honest question. Is one HTTP check on your homepage enough to know your whole site works?

For most teams it is not. Here is a real case that shows why.

The site was gone, but it answered 200 OK

I was looking at a startup's site last week and it would not load right. Not an error, not a timeout, just the wrong page. The domain had expired two days earlier.

When a domain lapses, the registrar does not switch the site off. It points the domain at its own parking page, the "this domain may be for sale" one. That page is a normal web page. It returns 200 OK and loads fast. It stays that way for a while too, because an expired domain is not deleted the next morning.

So the monitor sees a healthy site. Green light, no alert, everyone asleep. What could be wrong?

Your domain expired. You knew the date once, on the day you bought it a year or two ago. You told yourself you would renew closer to the time, and then you forgot. The card on file lapsed, or the reminder went to an inbox you no longer read.

Now your customers cannot get in. They panic. They email to ask whether you are still in business. And they get no answer, because the mail died with the domain too.

Why the mail dies with it

The registration carries your MX records, so when it lapses, mail to every address on the domain starts bouncing. The website looks up and the inbox goes silent, and the two failures do not obviously connect. That is why the angry email never reaches you.

The front page still answers. Behind it, nothing is running.

And the meter keeps running. While the page says 200, your ads keep sending paid clicks to a parking lander, signups never arrive, and recovery is not fully in your hands. Two days down is not a rounding error either, because downtime adds up faster than the percentages suggest. Once a domain enters redemption after expiry, getting it back can cost far more than a renewal, and the registry sets the clock, not you.

The clock after expiry

For a gTLD like a .com, the sequence is roughly this:

Grace period. The registrar parks the domain. The site is gone but the name still answers, usually for around 30 days, and a normal renewal still works.
Redemption, 30 days. Now it really goes dark. You can still get it back, but through a restore fee that is many times the renewal price.
Pending delete, about 5 days. Nothing you can do. At the end it drops to whoever is waiting.

ICANN's renewal and expiration FAQ and its grace period notes have the exact stages. The practical takeaway is that the first month, the one where an uptime monitor is happily reporting 200, is the cheap month. Everything after that costs money or the domain.

My own uptime check would miss it too

I build an uptime monitor, so here is the uncomfortable part. My own HTTP check would have called that dead site up too. A 200 is a 200. A status-code check cannot tell a real page from a parking page, and it should not pretend that it can.

That is the trap. One HTTP check on the homepage, the most common setup there is, is blind by design. The site answers, the code is green, and the part that pays your bills is broken behind it.

You can bolt on body matching, and it helps a bit. Assert that the response contains a string only your app renders, and a parking page fails it. But a keyword check is still guessing at the cause after the fact. The registration date is a fact you can read months in advance, so read it.

One check is not enough, set up several

So how do you avoid reading about your own outage in a customer's email? Not with one cleverer check. With a few plain ones, each watching a different way to fail. Start with the two that hand you a date instead of a surprise.

Domain expiry. Reads the registration record and warns you weeks before the day it lapses.
TLS certificate expiry. The same shape, a full outage with the date printed on it in advance.

Then cover the failures a homepage check never sees: a TCP check on your database port, a DNS check that reads the answer from outside your own network, and a heartbeat for the backup job that dies without a sound. The full list is in do I need an uptime monitor.

Six checks you trust beat one that only ever watches the front door.

How to set up domain expiry with an alert

Two minutes now buys you weeks of warning later. The shape is the same in any tool that supports it:

Add a monitor and choose the Domain expiry type.
Enter the registered domain, for example yourbusiness.com, not a subdomain.
Set the thresholds in days. A common choice is a warning at 30 days and a critical alert at 7.
Attach a notification channel you actually read. Telegram, Slack, PagerDuty, SMS, a webhook, or email all work. Pick the one that reaches you at night.

From then on it checks once a day, warns you when the date crosses your line, and reminds you as it gets close. No page-watching, no false green.

If you would rather not add a service for it, a cron job that runs whois and greps the expiry date will get you most of the way. The point is the date, not the tool. What you must not do is leave the only reminder in a registrar email going to a founder's old inbox.

Key takeaways

One HTTP check on the homepage is not enough. It stays green while the domain, the certificate, DNS, or a background job is broken.

An expired domain serves a parking page that returns 200 OK, so status-code monitors report it as healthy.

Email dies at the same time and even more quietly, because the MX records lapse with the registration.

You keep losing money the whole time: paid clicks to a dead page, lost signups, and a slow, costly recovery.

Add the checks that fail on a known date first, domain and TLS expiry, then the rest.

Common questions

Why does an expired domain still return 200 OK? The registrar parks it on a lander page instead of taking it offline. The old server is gone, but the parking page is a real web page that answers normally, so a check that only reads the status code sees success.

How long before an expired domain is gone for good? You usually get a month or two. Grace period, then a 30-day redemption where the site is dark but recoverable for a fee, then a short pending-delete stage before it is released to anyone.

What else breaks when a domain expires? Your email, at the same moment and more quietly. The MX records go with the registration, so mail starts bouncing while you are still looking at a page that loads.

Is a keyword check enough instead? It catches the parking page, which is better than nothing, but only once you are already down. The expiry date is knowable months ahead. Use both if you like, but do not skip the date.

I build Uptimepage because I wanted checks like this for my own services: domain and TLS expiry, HTTP, TCP, ping and DNS, heartbeats for background jobs, and browser flows for logins, from several regions. It is AGPL-3.0 open source, so you can run it on your own server if you prefer.

Now the question I actually want answered in the comments: what is the dumbest green check you have been burned by? I have seen a health endpoint that returned 200 with a hardcoded {"status":"ok"} while the database behind it was unreachable for six hours. I suspect yours is better.

How I stop one bad probe from waking you at 3 a.m.

Slim — Tue, 28 Jul 2026 14:55:48 +0000

TL;DR. The most common false alert in uptime monitoring is one probe location having a bad network day. I treated that as a design requirement from day one, so the checking pipeline has two gates: a region only counts as down after it fails the same check twice in a row, and an incident only opens when enough regions agree. A region that goes silent leaves the vote instead of counting as down. One bad location cannot page you. The original post carries the three figures below as live widgets you can drive yourself.

The alert that was nothing

The phone buzzes at 3 a.m. "Your API is down." You get up, open the laptop, and everything is green. The site was fine the whole time. One monitoring server, somewhere far away, had a bad network moment and sent an alert about nothing.

This is the false alert everyone in monitoring knows. It costs you sleep, and then it costs you something worse: the next alert feels less serious. The day a real outage comes, you look at your phone and think "probably nothing again."

I knew this failure mode before I wrote the first line of the scheduler, so it became a requirement, on the same level as "checks must run on time." The rule I started from: a single bad location must never be able to page a customer. The whole checking pipeline, from how probes report to how incidents open, is shaped by that rule.

The shape it took is two gates. A failure has to pass both before anyone gets paged.

Gate one: fail twice, from the same place

One failed check proves very little. Networks lose packets, routers restart, and sometimes a DNS answer arrives a second too late. All of that can make one check fail while your site is healthy.

So a region only counts as down after it fails the same check twice in a row. Not two failures somewhere in the system: two failures from that one region, back to back. A single blip resets to zero on the next good check.

One blip resets the counter. Two failures in a row from the same region earn that region a single vote.

The count is a setting on each monitor. Two is the default. For a very sensitive check you can set it to one, and for a noisy target you can raise it. Checks run on the monitor's own interval, so with a one-minute check the second failure arrives about a minute after the first. That minute buys you a lot of silence for a very small delay.

Gate two: the vote

Passing gate one gives a region exactly one vote. Nothing more.

Say you check from five regions and one of them has a bad ISP day. It fails twice in a row and votes "down." The other four keep passing. One vote against four is not enough, so nothing happens and nobody gets paged.

One region against four cannot page you. Three of five can.

The default rule is a majority: more than half of the reporting regions have to agree. You can change it per monitor. "Any" opens an incident on the first confirmed region, which is useful when you want the earliest possible signal and can accept some noise. "All" waits until every region agrees, for a service that only matters when it is unreachable everywhere. Or you pick a fixed number, like two regions out of whatever you assign.

A monitor checked from a single region behaves the same under every rule. The vote only starts to protect you when you add a second location, and it gets better with a third.

Silence is not failure

Here is the part that took the most care to get right.

The probe regions push their results to the control plane, the brain of the system. The brain never calls out to ask "are you alive?" during a vote. It just counts the results that arrived in the last few check cycles.

That gives silence a clear meaning. A region that lost its own connection cannot send anything, so it simply is not in the vote. It is not a down vote and not an up vote; the region is out until it reports again. The majority recalculates over the regions that still speak. Five regions where one goes dark becomes a vote of four.

A silent region is not a down vote and not an up vote. It leaves the vote until it reports again.

I think this is the only honest reading. A region that cannot reach the brain is telling you nothing about your site. Treating its silence as "down" would turn every probe outage on the monitoring side into a fake incident on yours.

And it works in both directions. Missing data never opens an incident, and it never closes one. An open incident only closes when the down votes fall below the threshold and at least one region shows a real run of passing checks. Recovery needs proof, the same way failure does.

Who watches the watchers

There is one gap left. If every region that covers your monitor goes dark, the vote has nobody in it. Your site could burn down and no incident would open, because no data means no votes.

For that case there is a separate signal, one level below the checks. Every probe sends a small check-in to the brain on its own schedule, a heartbeat that has nothing to do with your monitors. When the last live probe covering a monitor goes stale, you get a different message: "NO DATA: monitoring interrupted, no check results received." It is honest about what it knows: the service cannot see your site right now. It does not claim your site is down, because it has no idea.

When probing returns, you get a "monitoring RESUMED, receiving check results again" note, and the vote picks up where it left off.

One more detail I like. If a large share of all monitors goes silent at the same time, that pattern is almost never a thousand customer problems. It is one problem, mine. In that case the system alarms me and holds the customer notices, so my bad day does not become spam on yours.

What this means for your setup

Three practical points follow from this, whatever tool you run.

Check from more than one region. The vote cannot protect you with a single location. Two regions with the majority rule means both have to agree, which already kills the classic false alert. Three gives you two out of three, which is the right balance for most services.

Leave the confirmation count at two unless you have a reason. It is the difference between "a packet got lost" and "this endpoint is failing."

Pick the rule to match the monitor. A payment API deserves majority or even any. An internal tool nobody uses at night can wait for all. The setting is per monitor, so you do not have to choose one policy for everything.

The goal of all this machinery is boring: when your phone buzzes, it is real. Everything else is plumbing.

Common questions

What is a false alert in uptime monitoring?

A false alert says your site is down when it is not. The usual cause is a problem near the probe, not near your site: a bad network path, a busy datacenter, a DNS hiccup on the monitoring side. Your site answered fine for every real user the whole time.

How many regions need to agree before an incident opens?

By default, more than half of the regions that are reporting results. A region only joins the down side after it fails the same check twice in a row. You can change the rule per monitor: any single region, all regions, or a fixed count.

What happens when a probe region goes offline?

It leaves the vote. A region that sends no results is not counted as down and not counted as up. The vote recalculates over the regions that still report. Missing data alone never opens an incident.

Does a silent region close my open incident?

No. Closing needs real proof: the down votes must fall below the threshold and at least one region must show a run of passing checks. Silence is not proof of recovery, so an open incident stays open until real results come back.

Key takeaways

One failed check from one place proves nothing. Confirm it from the same region before that region gets a say.

Give each confirmed region one vote, then require agreement. A majority of reporting regions is a good default.

A silent region must leave the vote, not join the down side. Otherwise every probe outage becomes a fake incident.

Missing data must not close an incident either. Recovery needs a real run of passing checks.

Cover the all-dark case with a separate heartbeat and a message that says "we cannot see your site," not "your site is down."

Two regions kill the classic false alert. Three is the sweet spot for most services.

The three figures above are live widgets in the original post: you can change the confirmation count and the vote rule and watch where the line falls. The whole project is open source, so the scheduler and the incident writer are there to read.

How does your setup handle a probe that goes quiet? I have seen tools count silence as down, and I would like to know if anyone has a better reading than "it leaves the vote."

How I mapped my codebase for humans and AI agents

Slim — Thu, 23 Jul 2026 14:17:23 +0000

TL;DR. I asked an AI model to turn my codebase into three things: a one-page summary for me, a JSON file for the next AI agent, and an interactive map you can click. It worked well, but only after one boring step: check every number against the code. The model got the shape right and several counts wrong. You can see the live map.

The interactive map itself, with one flow lit up across the system. Open the live version and click any flow.

A big codebase does not fit in your head. Mine is about 146,000 lines of Rust across 31 modules. When you open a project that size, the first hour is just finding where things are.

AI agents have the same problem. Every time an agent starts a task, it reads many files to learn how the system fits together. It does this from zero, every time. That is slow, and it costs money.

So I tried something. I asked an AI model to read the whole codebase and write three things. One for me, one for the next agent, and one for anyone.

The three files

The first file is a one-page summary for a human. It lists the rules that must always hold, the main parts and what each one does, the path a request takes, and the traps that waste time. It is the page I wish existed on my first day.

The second file is a JSON map for the next AI agent. It is not written to be pretty. It lists the invariants with the file or test that enforces each one, a short recipe for each common task, the known traps, and the key files. When the next agent starts a task, it reads this first and skips an hour of searching.

The third file is an interactive map for everyone. It shows the parts as boxes in columns. You pick a flow, like "a scheduled check" or "an agent login", and the path lights up across the boxes with numbered steps. It is live here: the architecture map.

The three files. The JSON map is the source of truth, and the human page and the interactive map are both built from it.

One file is the source of truth

The JSON map holds every fact. The other two files are only views of it. Build the map first, and the human page and the interactive map cannot disagree.

The method, not one magic prompt

There is no single prompt that does this well. The result comes from four steps, in order.

The four steps. Everything hangs off step two, the map.

First, explore in parallel. One agent cannot read 146,000 lines in one go. I let several agents read different parts at the same time, then joined their notes. This is faster and it covers more.

Second, build the machine map first. The JSON for the agent is the source of truth. The human page and the interactive map are only views of it. So I build the JSON first and put every fact in one place.

Third, check every number in that map, before you build anything else. This is the step people skip, and it is the most important one. You verify once, at the source, so a wrong count cannot spread into the other files.

Fourth, render the views from the map. The human page and the interactive map both come from the same JSON, so they cannot disagree. Each view has one clear reader: the page is for a new engineer, and the map is for a visitor who has never seen the code.

The numbers the AI got wrong

The model was good at structure. It found the parts, the flows, and the rules. But it guessed numbers, and some guesses were wrong.

It said there were 13 alert channels. The real number is 14.

It said there were about 80 error codes. The real number is 155.

It said there were 19 blog posts. The real number is 18.

None of these are small. If you publish them, you look careless, and the next agent trusts a wrong map. I only found them because I checked each count against the source code.

Check the numbers, at the source

An AI model is strong at shape and words. It is weak at exact numbers. Verify every count in the map, once, before you build anything from it. Then check the ones that matter yourself.

The prompts

Here are the three prompts, in the order I ran them. Build the map first, check it, then make the views. Change the details for your own project, and keep the "verify against the source" line in the first one.

For the machine map, build this first:

Read my whole codebase. Write one JSON file for the next AI agent that
will add a feature. Include the invariants, and for each one name the file
or test that enforces it. Add a short recipe for each common task: the
goal, then the files to touch in order. Add the known traps and the key
files. Keep every path exact. Verify every count against the source code.
Do not invent numbers. This file is data for a machine, not prose.

For the human summary, built from the map:

From that verified JSON, write one self-contained HTML page that explains
the system to a new engineer. Include the rules that must always hold,
the main parts and what each does, the path a request takes, the path data
takes, and the traps that waste time. Rank the parts by size. Do not add
any number that is not already in the JSON.

For the interactive map, built from the same map:

From the same JSON, build one self-contained interactive HTML page. Show
the parts as boxes in columns. Show each flow as a numbered path that lights
up across the boxes when I click it. Keep all styles and scripts in the
page. No build step.

Why this is worth doing

The human page saved me time the next week. The map helps me explain the system in one screen. And the JSON is the part I did not expect to like. The next agent that touches this code reads a map first, so it starts from step one instead of step zero.

The next agent starts from step one, not step zero

A map file hands the next AI agent the rules, the tasks, and the traps up front. It reads one small file instead of re-reading the whole codebase every time.

Key takeaways

Build the machine map (JSON) first. The human page and the interactive map are only views of it, so they cannot drift.

An AI model is strong at structure and weak at numbers. Verify every count at the source before you build anything from it.

Write a map file for the next AI agent, so it starts from step one instead of re-reading the whole codebase.

Explore in parallel. Several agents reading different parts cover more than one agent reading everything.

Give each file one clear reader. It keeps the writing simple.

The model got real counts wrong, like 13 channels when the real number was 14. Do not publish a number you did not check.

If you want to see the output, open the interactive map. The whole project is open source, so you can read the real code behind every box.

If you have mapped a codebase for an AI agent: which format did the agent actually use, and how many numbers did the model get wrong on your run? Mine missed three. I would like to know if that is typical.

How to write incident status updates that build trust

Slim — Tue, 21 Jul 2026 13:42:58 +0000

TL;DR. During an outage your status page has a second job: holding trust while the fix is still in progress. Write plainly. Say what customers feel first, say what you are doing, and promise a time for the next update. Then keep that promise, even when there is no news. The four stages are investigating, identified, monitoring, and resolved.

The idea of an incident commander did not come from software. It came from wildfires. In 1970s California, a run of huge fires pushed fire chiefs to study why large responses fell apart. The cause surprised them. It was not too few firefighters or too little water. It was communication. Different teams used different words and no one shared a plan. The fix was a simple system with clear roles and clear language, now known as the Incident Command System, and on-call engineers still use a version of it today.

Your status page is the communication part of that system. When something breaks, the repair happens in your code. The trust happens on the status page. This post is about the second part: how to write updates that keep people calm while the first part is still in progress.

The words cost real money

An outage is expensive before you write a single word. In ITIC's 2024 survey of more than 1,000 companies, over 90% of mid-size and large firms said one hour of downtime costs them more than $300,000. For 41% of them, one hour costs between $1 million and $5 million. Uptime Institute's outage research points the same way: about one in five recent outages cost more than $1 million, and more than half cost over $100,000.

You do not need to be a bank for this to matter. If your store makes $2,000 a day, one bad hour during a sale can cost more than a slow afternoon, because sales are not spread evenly across the day. The size of the number changes. The shape of the problem does not.

Here is the part teams forget. Some of that cost is the downtime itself. The rest is trust, and trust is where writing helps. Many providers offer an SLA, a promise to stay up with a penalty if they miss it. The penalty is almost always a service credit, which is money back on your bill. A credit refunds your invoice, not your customer's bad afternoon. Good updates cannot bring the service back. They can protect the relationship while it is down.

The four stages of an incident

Most status pages, including the big public ones, use the same four stages. They come from a shared standard, so a customer who has read one status page already understands yours.

Stage	What it means	What to write
Investigating	You know something is wrong. You do not know why yet.	Name the impact and say you are looking into it.
Identified	You found the cause and are working on a fix.	Say what is wrong in plain words and that a fix is coming.
Monitoring	The fix is live. You are watching to be sure it holds.	Say the fix is in and you are checking that it works.
Resolved	The service is back and stable.	Say it is over, and thank people for waiting.

The order matters. Do not jump to identified because you have a hunch. If you say you found the cause and then change your story an hour later, every update after that is worth less. Move a stage forward only when the facts move with you.

The rules of a good update

Five rules cover almost every message.

Start with impact. Say what the customer feels before you talk about servers or databases. "Some payments are failing" is more useful than "we are seeing elevated 5xx errors on the API gateway." Only the first one means anything to a customer.

Share facts, not guesses. Say what you have confirmed. If something is still unknown, say that too. Writing "we do not yet know the cause" is fine. Inventing a cause to sound in control is not. This rule earns its place under pressure: the Uptime Institute's 2025 report found that about 40% of organizations had a major outage caused by human error in the last three years, and 85% of those came from someone not following a procedure. An incident is exactly when tired people guess.

Promise the next update. Give a real time, like "next update by 15:00 UTC" or "another update within 30 minutes." This one line does more than any apology. It tells people they can close the tab and get on with their day.

Add a timestamp and a time zone. "In 30 minutes" means nothing if the reader does not know when you posted. Every update should carry a clear time.

Keep it plain. Short sentences. No blame, no jargon, no backstory. A worried customer reads fast.

A bad update and a better one

Here is a bad update of the kind you have all seen:

We are aware of an issue and our team is working hard to resolve it as quickly as possible. We apologize for any inconvenience.

It sounds polite and says nothing. What is broken? Who is affected? When will I hear more? A better version answers those questions:

Investigating: some customers cannot log in. Sign-ups and password resets are affected too. Our team is looking into the cause now. Next update by 14:30 UTC.

Same length. Only one of them lets a customer decide what to do next.

How often to post

Set the rhythm in your first message and keep it. Every 30 to 60 minutes is a good starting point for a serious outage. The exact number matters less than the promise. If you said 30 minutes, post at 30 minutes, even when the only news is "still working, no change yet." Silence after a promised time reads as "they have lost control." A boring on-time update beats an exciting late one.

Planned maintenance is the same skill, calmer

Maintenance uses three stages: scheduled, in progress, and completed. The difference is that you have time to write it well in advance. Tell people what will happen, when it will happen, and whether they need to do anything. Mark it in progress while the work runs, and completed when it is done. The tone is calmer because nothing is on fire, but the shape is the same: impact, timing, next update.

The mistake almost everyone makes

It is not the wording. It is the second update.

The first update is easy. The outage just happened, and everyone is paying attention. Then the fix takes longer than expected, the team goes quiet, and the page sits at "investigating" for two hours. From the outside, a frozen status page looks the same as a dead company.

The fix is a habit, not a talent. Decide the next-update time in your first message, set a timer, and post again when it rings. If there is no news, that is the update: no change, still working, next check in 30 minutes. The point of the promise is that people stop refreshing and trust you to come back.

Write the message, then give it a home

You can write all of this by hand during an outage. Most people write worse under stress, which is the worst possible time to start from a blank box. That is why we built a free incident update generator: pick a stage, fill in a few fields, and it writes a clean message you can paste into any status page. It runs in your browser and stores nothing you type.

The message still needs somewhere to live. A status page is where customers look when your product will not load, so it should be honest and easy to reach. If you are choosing an uptime number to promise, the uptime SLA calculator turns a percentage into real minutes, and is 98% uptime good shows how small percentages hide large amounts of downtime. For the page itself, a status page you cannot fake covers how to keep it worth trusting.

Common questions

What should the first incident update say? Say what the customer feels, say you are looking into it, and give a time for the next update. You do not need the cause yet. Waiting for the root cause before posting is the most common mistake.

What is the difference between investigating and identified? Investigating means the cause is not confirmed. Identified means you have confirmed the cause and are working on a fix. Do not move to identified based on a hunch.

Should you apologize in an incident update? One short, honest line is fine. Skip the long apology. A clear next-update time does more for trust than three sentences of sorry.

If you have run incidents: what update cadence actually held up for you under a long outage, and did anyone ever complain that you posted too often? I have never seen that complaint, but I would like to know if it exists.

Why I chose Rust over Go for an uptime monitor

Slim — Mon, 20 Jul 2026 14:46:30 +0000

I build Uptimepage, an open-source uptime monitor and status page written in Rust. People ask why Rust and not Go, since Go is the usual pick for this kind of network service. Here is the honest answer. It is not that Rust wins everywhere. It is that one part of this job made the choice for me.

The product is one promise: tell you fast and honestly when your site is slow or down. The numbers I show you, like your p99 response time, have to be clean. If my own code adds random delay, I blur the exact signal you pay for. So the runtime under the prober matters more here than it would for a normal app.

Garbage collection shows up in the tail

Go has a garbage collector. It is fast, and most apps never feel it. But it still has to do work to free memory, and that work can land inside the millisecond timings I report. Run tens of thousands of checks at once and a pause at the wrong moment lifts a p99 number. At that point I am measuring my own runtime, not your server.

Rust has no garbage collector. Memory is freed at a point I can see in the code. There is no background pause I did not write. For a tool that sells timing, that control is worth the extra work Rust asks for.

What that looks like in memory

Here are some real numbers, with a warning attached. They come from a load test on a developer laptop, not a production server. I use them to catch a slowdown between two versions of my code, not to plan capacity. A real server does better, a small box does worse. Treat them as a floor.

In one run, a single machine held 50,000 checks in flight and peaked at 933 MiB of memory. That is under one gigabyte for fifty thousand live checks. My running server uses about 42 MiB while watching its monitors, and it sits quiet when there is nothing to do. That kind of density is normal for a service with no garbage collector, and it means one small box covers a lot of monitors. I go deeper on the prober and the throughput numbers in the build story.

The bug that never compiles

Speed is only half of it. The other half is a class of bug that Rust refuses to build.

Picture many workers writing to one shared map of results at the same time. In Go this compiles and runs. Sometimes it is fine. Sometimes two goroutines write at once, you get corrupt data or a crash, and it only happens under load, which is the worst time to find out:

// compiles fine, races at runtime
results := map[string]int{}
for _, check := range checks {
    go func(c Check) {
        results[c.ID] = c.Latency // data race
    }(check)
}

Go gives you good tools for this. There is a race detector, a sync.Mutex, and channels. But remembering to reach for them is on you.

In Rust the same concurrent write does not compile. The compiler stops you until the shared map is wrapped in a lock:

// will not compile unless the shared map is a Mutex
let results = Mutex::new(HashMap::new());
std::thread::scope(|s| {
    for check in checks {
        let results = &results;
        s.spawn(move || {
            // each worker locks only for its own insert
            results.lock().unwrap().insert(check.id, check.latency);
        });
    }
});

Each worker holds the lock only for its own insert, so the writes stay safe without blocking the others for long. For code that runs day and night across many machines, "the compiler will not let you ship the race" removes a whole set of late-night bugs before they exist.

Where Go is the better pick

None of this makes Go a bad choice. Often it is the better one. Go is faster to learn. It builds in seconds. Its standard library for network services is excellent, and a new engineer can be useful in days. If I were building a normal web service, or something I had to ship this week, Go would be on the table and might win.

Rust asks more from you first. The compiler argues with you. The build is slower. You spend time on things Go would just handle. I take that trade because this job is narrow and it rewards tight control over memory and timing.

The compiler as a safety net

Heavy code review is how large teams catch data races and memory bugs. Rust gives you a lot of that for free. The compiler turns those mistakes into build errors, so they never reach production and never wake anyone up. Every change gets a strong, automatic check before it ships, which is a big part of why I trust the service to run unattended.

It is also why Uptimepage is open source and self-hostable. You can read the code, run it on your own hardware, and export your data whenever you want, so you are never locked into a single vendor. You can start from GitHub.

The honest version

The honest version is not "Rust beats Go." It is "for a tool that lives or dies by timing and runs at high concurrency, Rust fit better." Pick the language for the job in front of you. Mine happened to be a job that Rust is very good at.

If you have shipped a heavily concurrent network service, did you reach for Go or Rust, and did the GC ever actually show up in your tail latencies? Curious where people land when the timing is the product.

Is 98% uptime good? It allows 7.3 days of downtime a year

Slim — Fri, 17 Jul 2026 13:26:37 +0000

Cover photo by the blowup on Unsplash.

I build an uptime monitor, so this question lands in my inbox a lot: is 98% uptime good?

For a public website or a paid API, no. For an internal tool or a side project, it is fine. The difference is one division away.

98% looks like a top grade because school taught us that 98 out of 100 is excellent. Uptime does not grade like school. The whole scale for public services lives between 99% and 100%, and serious targets differ only in the digits after the decimal. On that scale, 98% sits at the bottom.

Turn the percentage into time

The allowed failure at 98% is 2%. Two percent of a year is 7.3 days. Two percent of a 30-day month is 14.4 hours. If your shop makes $2,000 a day, 98% uptime means you accept about $14,600 of closed-door time per year and the target still counts as met.

Here is the same math for every common target. The last column prices the downtime for that $2,000-a-day shop.

Uptime	Per day	Per 30-day month	Per year	Lost sales per year
90%	2.4 hours	3 days	36.5 days	$73,000
95%	1.2 hours	36 hours	18.3 days	$36,500
98%	28.8 minutes	14.4 hours	7.3 days	$14,600
99%	14.4 minutes	7.2 hours	3.7 days	$7,300
99.5%	7.2 minutes	3.6 hours	1.8 days	$3,650
99.9%	1.4 minutes	43 minutes	8.8 hours	$730
99.95%	43 seconds	21.6 minutes	4.4 hours	$365
99.99%	8.6 seconds	4.3 minutes	52.6 minutes	$73
99.999%	0.9 seconds	26 seconds	5.3 minutes	$7

Two jumps on this table do most of the work in real contracts. From 98% to 99.9%, the allowed downtime drops from 14.4 hours a month to 43 minutes. From 99.9% to 99.99%, it drops from 43 minutes to 4.3 minutes, and that second jump usually costs about ten times more engineering than the first while saving the shop $657 a year.

The shape of the downtime matters

98% per month is 14.4 hours, but the number says nothing about how those hours land.

Thirty minutes of planned maintenance every night at 03:00 adds up to 98% and most users never notice. One 14-hour outage on the day of your product launch is also 98%. Same score, very different month.

So a single uptime percentage is a summary, not the full story. When someone quotes you a number, also ask about the longest single outage and when it happened.

When 98% is enough

Plenty of systems can live at 98% and nobody gets hurt:

An internal wiki. People retry after lunch.
A staging environment. Downtime there is often planned.
A batch job that builds reports at night. It has hours of slack before anyone reads the output.
A home server on a residential connection. Your power company already decided your uptime for you.

The shared pattern: when these go down, nobody loses money and nobody loses trust. Paying for more nines there is waste.

When it is not

A checkout page, a paid API, a login service. Here 98% fails twice. First the direct cost: 14.4 hours a month of failed requests and support tickets. Second the trust cost, which is larger and slower. A customer who hits your outage twice in one week does not check your uptime report. They remember that your service is the one that breaks.

One more trap: an SLA is not uptime. Uptime is the measured number. An SLA is a contract promise with a penalty, and the penalty is almost always a service credit. If a provider misses its 99.9% SLA, you get part of your bill back. Your customers get nothing.

What to aim for instead

99.9% is the default target for customer-facing services for a reason. 43 minutes a month is enough room for a bad deploy and a couple of small failures, and a small team can hit it without heroics. Above that, each nine costs roughly ten times more and most users cannot feel the difference.

If you want to run your own numbers, I keep a free uptime SLA calculator and an error budget calculator on our site; both work without signup.

What target do you actually run in production, and did you pick it or inherit it? I am curious how many teams measured before they promised.

The status page you can't fake: measured uptime, not published

Slim — Wed, 15 Jul 2026 13:50:26 +0000

A status page is the one dashboard a company publishes about its own service. It is also the one place where the company has a reason to look good. That is a problem. If the page can be edited to look better than reality, it stops being useful. So when you build or choose a status page, ask one thing: can someone hide a real outage on it?

The short answer: a status page you can trust builds its uptime bar from real checks, not from the incidents a person chose to publish.

Two layers, and only one is yours to edit

A good status page has two parts. The first part is the measured data: the green and red timeline, the 90-day bar, the uptime percent. The second part is incidents: the notes a person writes to explain what broke, what they are doing, and when it is fixed. On the page they sit next to each other and look the same. They are not the same, and mixing them is where trust gets lost.

The measured data answers one question: what did the checks see? The incident notes answer a different one: what does the team want to say about it? The first is a fact. The second is a story. A status page you can trust lets the team write the story, but keeps them away from the facts.

The story layer also includes the postmortem, the write-up you post after an outage. A postmortem is honesty you add on purpose. You explain what broke and why, because you choose to. The bar works the other way. It shows the failure on its own, whether you write anything or not. You control the story. You do not control the facts.

The trap: a bar that only shows what you published

The common mistake is to build the uptime bar from incidents. It feels natural. You already open an incident when something breaks, so why not color the timeline from incidents too? Now the bar turns red only where an incident exists and is marked public.

The problem comes on the day an outage has no public incident. Maybe no one published it. Maybe a setting was wrong. Maybe the monitor was added to the page after the outage, so its incident was saved as private and never checked again. In every case the timeline shows green over a real red day, and it does this quietly. The uptime number goes up. The customer sees 100 percent over a week they remember as broken.

One version of this trap is easy to build by accident, so it is worth explaining. You decide "is this incident public?" once, at the moment the incident opens, based on whether the monitor was on a public page right then. Then you save that answer and never look again. In the code it looks like a live check. It is really a photo taken one time. Move the monitor onto the page a day later, and its past outages stay hidden, because the photo was taken before the monitor was there. Any yes or no mark that is set once and then trusted forever has this problem.

The fix is a rule, not a switch

The bar should come from the measured data. It needs only one rule to stay calm: wait for confirmation before you count downtime. You do not want one failed check from one place to turn a whole day red, because networks are noisy and one bad checker is not an outage. So you wait for agreement: more than one region failing at the same time, for more than one check in a row. That one rule is enough to keep a short blip from becoming a red day.

One thing matters here: the same rule feeds both your alerts and your uptime bar. If the rule that wakes your on-call person is the same rule that colors the timeline, the two can never tell different stories. The moment you add a second rule just for the bar, it will drift from the first, and the page will disagree with itself. On-call gets paged, but the public history says everything was fine.

Publishing stays where it belongs, on the notes. The team decides whether to write an incident, what to say, and when to post the all-clear. They do not decide whether last Tuesday was down. The checks already decided that.

Test the status page you already have

You can check any status page, including your own, in a few minutes.

Add a monitor to a page after it has already had an outage. Does the history show the outage, or does it start clean from the day you added the monitor?
Take a real incident and unpublish it. Does the bar keep the red day, or does the day turn green?
Make one region fail for one second. Does the whole day go red, or does the bar stay calm?

A page you can trust shows the outage in the first test, keeps the red in the second, and stays calm in the third. A page that fails these is usually not lying on purpose. It just built its timeline on top of what people chose to publish, and that always has holes.

Where this leaves you

This is how I built it into Uptimepage: the 90-day bar and each status light come from confirmed downtime, measured across regions with a confirmation rule, not from what someone chose to publish. Incidents and postmortems are the layer you write by hand. It is AGPL and open source on GitHub, and there is a longer write-up on the internals if you want the Rust side.

You should not be able to fake your uptime. You should not be able to fake it by accident either. The bar is a measurement. Keep it one.

How does your status page compute its uptime bar, from checks or from published incidents? Curious what people run and whether anyone has been bitten by the frozen-flag version of this.

Error budgets, explained: SLOs, burn rate, and when to stop shipping

Slim — Mon, 13 Jul 2026 14:11:33 +0000

The idea

An uptime target has a second number hidden inside it. Say you promise 99.9% uptime. You are also saying that 0.1% is allowed to fail. Put that 0.1% into real time, and that is your error budget: the downtime you can have before you break the promise.

This changes how you look at downtime. It stops being a mistake to feel bad about and becomes a budget you can spend: on a risky deploy, on a slow service you depend on, or on a migration. When the budget runs low you slow down. When it is healthy you can move fast.

The formula

An SLO is your reliability target, for example 99.9%. The gap to 100% is the failure you are allowed. Multiply that gap by the length of the window, and you get real time you can spend.

error budget = window x (1 - SLO)

A 99.9% SLO over a 30-day month is 2,592,000 seconds times 0.001. That is 43 minutes and 12 seconds. That is the whole budget for the month, shared across every incident, not a fresh 43 minutes each day.

So three nines is not "never go down". It is a 43-minute budget each month. Every extra nine costs about ten times more engineering, for downtime that most users never notice. This is why Google says that 100% is the wrong target for almost every service.

Burn rate is the speed

The total tells you how much you can spend. It does not tell you how fast. Two services can both sit at 99.9% for the month: one loses the budget slowly, the other loses it all in a single bad hour. Burn rate tells them apart.

burn rate = (1 - measured) / (1 - SLO)

A burn rate of 1x spends the whole window exactly. 2x spends it in half the time. Below 1x, you finish the month with budget left. Above 1x, the rate tells you the deadline:

budget runs out in = window / burn rate

At 2x, a 30-day budget is gone in fifteen days. At 14.4x it is gone in about two days. That same 14.4x spends 2% of the budget in one hour, which is the level most fast alerts are set to.

Turning burn rate into alerts

One threshold is not enough. It either alerts too late or it sends too many false alarms. The common fix uses two windows: a long one to confirm the problem is real, and a short one to clear the alert quickly once you fix it. Both have to be burning for the alert to fire. For a 30-day budget, these are the usual settings:

Fast page: 2% of the budget in 1 hour (with a 5-minute short window). This is a 14.4x burn.
Page: 5% in 6 hours (30-minute short window). This is a 6x burn.
Slow ticket: 10% in 3 days (6-hour short window). This is a 1x burn.

The fast page catches a sudden outage. The slow ticket catches a slow problem that would still use up the whole month if nobody looked.

The rule is the point

The math is the easy part. The value comes from a rule you agree on before anything breaks. The rule is simple. When the budget runs out, risky launches stop, and the team works on reliability until the budget grows back. While the budget is healthy, you ship and you take the risk.

One more rule keeps it fair. If you never spend your budget, your SLO is too strict, and you are paying for reliability that no user asked for.

Try it

I put the formulas into a free, no sign-up error budget calculator: enter an SLO and your measured availability, and it shows budget spent, budget left, burn rate, and a burn-down chart. There is also an uptime SLA calculator for the full downtime-per-nine table.

What SLO and burn-rate thresholds do you run in production? Curious how others pick them.