DEV Community: Slim

How I stop one bad probe from waking you at 3 a.m.

Slim — Tue, 28 Jul 2026 14:55:48 +0000

TL;DR. The most common false alert in uptime monitoring is one probe location having a bad network day. I treated that as a design requirement from day one, so the checking pipeline has two gates: a region only counts as down after it fails the same check twice in a row, and an incident only opens when enough regions agree. A region that goes silent leaves the vote instead of counting as down. One bad location cannot page you. The original post carries the three figures below as live widgets you can drive yourself.

The alert that was nothing

The phone buzzes at 3 a.m. "Your API is down." You get up, open the laptop, and everything is green. The site was fine the whole time. One monitoring server, somewhere far away, had a bad network moment and sent an alert about nothing.

This is the false alert everyone in monitoring knows. It costs you sleep, and then it costs you something worse: the next alert feels less serious. The day a real outage comes, you look at your phone and think "probably nothing again."

I knew this failure mode before I wrote the first line of the scheduler, so it became a requirement, on the same level as "checks must run on time." The rule I started from: a single bad location must never be able to page a customer. The whole checking pipeline, from how probes report to how incidents open, is shaped by that rule.

The shape it took is two gates. A failure has to pass both before anyone gets paged.

Gate one: fail twice, from the same place

One failed check proves very little. Networks lose packets, routers restart, and sometimes a DNS answer arrives a second too late. All of that can make one check fail while your site is healthy.

So a region only counts as down after it fails the same check twice in a row. Not two failures somewhere in the system: two failures from that one region, back to back. A single blip resets to zero on the next good check.

One blip resets the counter. Two failures in a row from the same region earn that region a single vote.

The count is a setting on each monitor. Two is the default. For a very sensitive check you can set it to one, and for a noisy target you can raise it. Checks run on the monitor's own interval, so with a one-minute check the second failure arrives about a minute after the first. That minute buys you a lot of silence for a very small delay.

Gate two: the vote

Passing gate one gives a region exactly one vote. Nothing more.

Say you check from five regions and one of them has a bad ISP day. It fails twice in a row and votes "down." The other four keep passing. One vote against four is not enough, so nothing happens and nobody gets paged.

One region against four cannot page you. Three of five can.

The default rule is a majority: more than half of the reporting regions have to agree. You can change it per monitor. "Any" opens an incident on the first confirmed region, which is useful when you want the earliest possible signal and can accept some noise. "All" waits until every region agrees, for a service that only matters when it is unreachable everywhere. Or you pick a fixed number, like two regions out of whatever you assign.

A monitor checked from a single region behaves the same under every rule. The vote only starts to protect you when you add a second location, and it gets better with a third.

Silence is not failure

Here is the part that took the most care to get right.

The probe regions push their results to the control plane, the brain of the system. The brain never calls out to ask "are you alive?" during a vote. It just counts the results that arrived in the last few check cycles.

That gives silence a clear meaning. A region that lost its own connection cannot send anything, so it simply is not in the vote. It is not a down vote and not an up vote; the region is out until it reports again. The majority recalculates over the regions that still speak. Five regions where one goes dark becomes a vote of four.

A silent region is not a down vote and not an up vote. It leaves the vote until it reports again.

I think this is the only honest reading. A region that cannot reach the brain is telling you nothing about your site. Treating its silence as "down" would turn every probe outage on the monitoring side into a fake incident on yours.

And it works in both directions. Missing data never opens an incident, and it never closes one. An open incident only closes when the down votes fall below the threshold and at least one region shows a real run of passing checks. Recovery needs proof, the same way failure does.

Who watches the watchers

There is one gap left. If every region that covers your monitor goes dark, the vote has nobody in it. Your site could burn down and no incident would open, because no data means no votes.

For that case there is a separate signal, one level below the checks. Every probe sends a small check-in to the brain on its own schedule, a heartbeat that has nothing to do with your monitors. When the last live probe covering a monitor goes stale, you get a different message: "NO DATA: monitoring interrupted, no check results received." It is honest about what it knows: the service cannot see your site right now. It does not claim your site is down, because it has no idea.

When probing returns, you get a "monitoring RESUMED, receiving check results again" note, and the vote picks up where it left off.

One more detail I like. If a large share of all monitors goes silent at the same time, that pattern is almost never a thousand customer problems. It is one problem, mine. In that case the system alarms me and holds the customer notices, so my bad day does not become spam on yours.

What this means for your setup

Three practical points follow from this, whatever tool you run.

Check from more than one region. The vote cannot protect you with a single location. Two regions with the majority rule means both have to agree, which already kills the classic false alert. Three gives you two out of three, which is the right balance for most services.

Leave the confirmation count at two unless you have a reason. It is the difference between "a packet got lost" and "this endpoint is failing."

Pick the rule to match the monitor. A payment API deserves majority or even any. An internal tool nobody uses at night can wait for all. The setting is per monitor, so you do not have to choose one policy for everything.

The goal of all this machinery is boring: when your phone buzzes, it is real. Everything else is plumbing.

Common questions

What is a false alert in uptime monitoring?

A false alert says your site is down when it is not. The usual cause is a problem near the probe, not near your site: a bad network path, a busy datacenter, a DNS hiccup on the monitoring side. Your site answered fine for every real user the whole time.

How many regions need to agree before an incident opens?

By default, more than half of the regions that are reporting results. A region only joins the down side after it fails the same check twice in a row. You can change the rule per monitor: any single region, all regions, or a fixed count.

What happens when a probe region goes offline?

It leaves the vote. A region that sends no results is not counted as down and not counted as up. The vote recalculates over the regions that still report. Missing data alone never opens an incident.

Does a silent region close my open incident?

No. Closing needs real proof: the down votes must fall below the threshold and at least one region must show a run of passing checks. Silence is not proof of recovery, so an open incident stays open until real results come back.

Key takeaways

One failed check from one place proves nothing. Confirm it from the same region before that region gets a say.

Give each confirmed region one vote, then require agreement. A majority of reporting regions is a good default.

A silent region must leave the vote, not join the down side. Otherwise every probe outage becomes a fake incident.

Missing data must not close an incident either. Recovery needs a real run of passing checks.

Cover the all-dark case with a separate heartbeat and a message that says "we cannot see your site," not "your site is down."

Two regions kill the classic false alert. Three is the sweet spot for most services.

The three figures above are live widgets in the original post: you can change the confirmation count and the vote rule and watch where the line falls. The whole project is open source, so the scheduler and the incident writer are there to read.

How does your setup handle a probe that goes quiet? I have seen tools count silence as down, and I would like to know if anyone has a better reading than "it leaves the vote."

How I mapped my codebase for humans and AI agents

Slim — Thu, 23 Jul 2026 14:17:23 +0000

TL;DR. I asked an AI model to turn my codebase into three things: a one-page summary for me, a JSON file for the next AI agent, and an interactive map you can click. It worked well, but only after one boring step: check every number against the code. The model got the shape right and several counts wrong. You can see the live map.

The interactive map itself, with one flow lit up across the system. Open the live version and click any flow.

A big codebase does not fit in your head. Mine is about 146,000 lines of Rust across 31 modules. When you open a project that size, the first hour is just finding where things are.

AI agents have the same problem. Every time an agent starts a task, it reads many files to learn how the system fits together. It does this from zero, every time. That is slow, and it costs money.

So I tried something. I asked an AI model to read the whole codebase and write three things. One for me, one for the next agent, and one for anyone.

The three files

The first file is a one-page summary for a human. It lists the rules that must always hold, the main parts and what each one does, the path a request takes, and the traps that waste time. It is the page I wish existed on my first day.

The second file is a JSON map for the next AI agent. It is not written to be pretty. It lists the invariants with the file or test that enforces each one, a short recipe for each common task, the known traps, and the key files. When the next agent starts a task, it reads this first and skips an hour of searching.

The third file is an interactive map for everyone. It shows the parts as boxes in columns. You pick a flow, like "a scheduled check" or "an agent login", and the path lights up across the boxes with numbered steps. It is live here: the architecture map.

The three files. The JSON map is the source of truth, and the human page and the interactive map are both built from it.

One file is the source of truth

The JSON map holds every fact. The other two files are only views of it. Build the map first, and the human page and the interactive map cannot disagree.

The method, not one magic prompt

There is no single prompt that does this well. The result comes from four steps, in order.

The four steps. Everything hangs off step two, the map.

First, explore in parallel. One agent cannot read 146,000 lines in one go. I let several agents read different parts at the same time, then joined their notes. This is faster and it covers more.

Second, build the machine map first. The JSON for the agent is the source of truth. The human page and the interactive map are only views of it. So I build the JSON first and put every fact in one place.

Third, check every number in that map, before you build anything else. This is the step people skip, and it is the most important one. You verify once, at the source, so a wrong count cannot spread into the other files.

Fourth, render the views from the map. The human page and the interactive map both come from the same JSON, so they cannot disagree. Each view has one clear reader: the page is for a new engineer, and the map is for a visitor who has never seen the code.

The numbers the AI got wrong

The model was good at structure. It found the parts, the flows, and the rules. But it guessed numbers, and some guesses were wrong.

It said there were 13 alert channels. The real number is 14.

It said there were about 80 error codes. The real number is 155.

It said there were 19 blog posts. The real number is 18.

None of these are small. If you publish them, you look careless, and the next agent trusts a wrong map. I only found them because I checked each count against the source code.

Check the numbers, at the source

An AI model is strong at shape and words. It is weak at exact numbers. Verify every count in the map, once, before you build anything from it. Then check the ones that matter yourself.

The prompts

Here are the three prompts, in the order I ran them. Build the map first, check it, then make the views. Change the details for your own project, and keep the "verify against the source" line in the first one.

For the machine map, build this first:

Read my whole codebase. Write one JSON file for the next AI agent that
will add a feature. Include the invariants, and for each one name the file
or test that enforces it. Add a short recipe for each common task: the
goal, then the files to touch in order. Add the known traps and the key
files. Keep every path exact. Verify every count against the source code.
Do not invent numbers. This file is data for a machine, not prose.

For the human summary, built from the map:

From that verified JSON, write one self-contained HTML page that explains
the system to a new engineer. Include the rules that must always hold,
the main parts and what each does, the path a request takes, the path data
takes, and the traps that waste time. Rank the parts by size. Do not add
any number that is not already in the JSON.

For the interactive map, built from the same map:

From the same JSON, build one self-contained interactive HTML page. Show
the parts as boxes in columns. Show each flow as a numbered path that lights
up across the boxes when I click it. Keep all styles and scripts in the
page. No build step.

Why this is worth doing

The human page saved me time the next week. The map helps me explain the system in one screen. And the JSON is the part I did not expect to like. The next agent that touches this code reads a map first, so it starts from step one instead of step zero.

The next agent starts from step one, not step zero

A map file hands the next AI agent the rules, the tasks, and the traps up front. It reads one small file instead of re-reading the whole codebase every time.

Key takeaways

Build the machine map (JSON) first. The human page and the interactive map are only views of it, so they cannot drift.

An AI model is strong at structure and weak at numbers. Verify every count at the source before you build anything from it.

Write a map file for the next AI agent, so it starts from step one instead of re-reading the whole codebase.

Explore in parallel. Several agents reading different parts cover more than one agent reading everything.

Give each file one clear reader. It keeps the writing simple.

The model got real counts wrong, like 13 channels when the real number was 14. Do not publish a number you did not check.

If you want to see the output, open the interactive map. The whole project is open source, so you can read the real code behind every box.

If you have mapped a codebase for an AI agent: which format did the agent actually use, and how many numbers did the model get wrong on your run? Mine missed three. I would like to know if that is typical.

How to write incident status updates that build trust

Slim — Tue, 21 Jul 2026 13:42:58 +0000

TL;DR. During an outage your status page has a second job: holding trust while the fix is still in progress. Write plainly. Say what customers feel first, say what you are doing, and promise a time for the next update. Then keep that promise, even when there is no news. The four stages are investigating, identified, monitoring, and resolved.

The idea of an incident commander did not come from software. It came from wildfires. In 1970s California, a run of huge fires pushed fire chiefs to study why large responses fell apart. The cause surprised them. It was not too few firefighters or too little water. It was communication. Different teams used different words and no one shared a plan. The fix was a simple system with clear roles and clear language, now known as the Incident Command System, and on-call engineers still use a version of it today.

Your status page is the communication part of that system. When something breaks, the repair happens in your code. The trust happens on the status page. This post is about the second part: how to write updates that keep people calm while the first part is still in progress.

The words cost real money

An outage is expensive before you write a single word. In ITIC's 2024 survey of more than 1,000 companies, over 90% of mid-size and large firms said one hour of downtime costs them more than $300,000. For 41% of them, one hour costs between $1 million and $5 million. Uptime Institute's outage research points the same way: about one in five recent outages cost more than $1 million, and more than half cost over $100,000.

You do not need to be a bank for this to matter. If your store makes $2,000 a day, one bad hour during a sale can cost more than a slow afternoon, because sales are not spread evenly across the day. The size of the number changes. The shape of the problem does not.

Here is the part teams forget. Some of that cost is the downtime itself. The rest is trust, and trust is where writing helps. Many providers offer an SLA, a promise to stay up with a penalty if they miss it. The penalty is almost always a service credit, which is money back on your bill. A credit refunds your invoice, not your customer's bad afternoon. Good updates cannot bring the service back. They can protect the relationship while it is down.

The four stages of an incident

Most status pages, including the big public ones, use the same four stages. They come from a shared standard, so a customer who has read one status page already understands yours.

Stage	What it means	What to write
Investigating	You know something is wrong. You do not know why yet.	Name the impact and say you are looking into it.
Identified	You found the cause and are working on a fix.	Say what is wrong in plain words and that a fix is coming.
Monitoring	The fix is live. You are watching to be sure it holds.	Say the fix is in and you are checking that it works.
Resolved	The service is back and stable.	Say it is over, and thank people for waiting.

The order matters. Do not jump to identified because you have a hunch. If you say you found the cause and then change your story an hour later, every update after that is worth less. Move a stage forward only when the facts move with you.

The rules of a good update

Five rules cover almost every message.

Start with impact. Say what the customer feels before you talk about servers or databases. "Some payments are failing" is more useful than "we are seeing elevated 5xx errors on the API gateway." Only the first one means anything to a customer.

Share facts, not guesses. Say what you have confirmed. If something is still unknown, say that too. Writing "we do not yet know the cause" is fine. Inventing a cause to sound in control is not. This rule earns its place under pressure: the Uptime Institute's 2025 report found that about 40% of organizations had a major outage caused by human error in the last three years, and 85% of those came from someone not following a procedure. An incident is exactly when tired people guess.

Promise the next update. Give a real time, like "next update by 15:00 UTC" or "another update within 30 minutes." This one line does more than any apology. It tells people they can close the tab and get on with their day.

Add a timestamp and a time zone. "In 30 minutes" means nothing if the reader does not know when you posted. Every update should carry a clear time.

Keep it plain. Short sentences. No blame, no jargon, no backstory. A worried customer reads fast.

A bad update and a better one

Here is a bad update of the kind you have all seen:

We are aware of an issue and our team is working hard to resolve it as quickly as possible. We apologize for any inconvenience.

It sounds polite and says nothing. What is broken? Who is affected? When will I hear more? A better version answers those questions:

Investigating: some customers cannot log in. Sign-ups and password resets are affected too. Our team is looking into the cause now. Next update by 14:30 UTC.

Same length. Only one of them lets a customer decide what to do next.

How often to post

Set the rhythm in your first message and keep it. Every 30 to 60 minutes is a good starting point for a serious outage. The exact number matters less than the promise. If you said 30 minutes, post at 30 minutes, even when the only news is "still working, no change yet." Silence after a promised time reads as "they have lost control." A boring on-time update beats an exciting late one.

Planned maintenance is the same skill, calmer

Maintenance uses three stages: scheduled, in progress, and completed. The difference is that you have time to write it well in advance. Tell people what will happen, when it will happen, and whether they need to do anything. Mark it in progress while the work runs, and completed when it is done. The tone is calmer because nothing is on fire, but the shape is the same: impact, timing, next update.

The mistake almost everyone makes

It is not the wording. It is the second update.

The first update is easy. The outage just happened, and everyone is paying attention. Then the fix takes longer than expected, the team goes quiet, and the page sits at "investigating" for two hours. From the outside, a frozen status page looks the same as a dead company.

The fix is a habit, not a talent. Decide the next-update time in your first message, set a timer, and post again when it rings. If there is no news, that is the update: no change, still working, next check in 30 minutes. The point of the promise is that people stop refreshing and trust you to come back.

Write the message, then give it a home

You can write all of this by hand during an outage. Most people write worse under stress, which is the worst possible time to start from a blank box. That is why we built a free incident update generator: pick a stage, fill in a few fields, and it writes a clean message you can paste into any status page. It runs in your browser and stores nothing you type.

The message still needs somewhere to live. A status page is where customers look when your product will not load, so it should be honest and easy to reach. If you are choosing an uptime number to promise, the uptime SLA calculator turns a percentage into real minutes, and is 98% uptime good shows how small percentages hide large amounts of downtime. For the page itself, a status page you cannot fake covers how to keep it worth trusting.

Common questions

What should the first incident update say? Say what the customer feels, say you are looking into it, and give a time for the next update. You do not need the cause yet. Waiting for the root cause before posting is the most common mistake.

What is the difference between investigating and identified? Investigating means the cause is not confirmed. Identified means you have confirmed the cause and are working on a fix. Do not move to identified based on a hunch.

Should you apologize in an incident update? One short, honest line is fine. Skip the long apology. A clear next-update time does more for trust than three sentences of sorry.

If you have run incidents: what update cadence actually held up for you under a long outage, and did anyone ever complain that you posted too often? I have never seen that complaint, but I would like to know if it exists.

Why I chose Rust over Go for an uptime monitor

Slim — Mon, 20 Jul 2026 14:46:30 +0000

I build Uptimepage, an open-source uptime monitor and status page written in Rust. People ask why Rust and not Go, since Go is the usual pick for this kind of network service. Here is the honest answer. It is not that Rust wins everywhere. It is that one part of this job made the choice for me.

The product is one promise: tell you fast and honestly when your site is slow or down. The numbers I show you, like your p99 response time, have to be clean. If my own code adds random delay, I blur the exact signal you pay for. So the runtime under the prober matters more here than it would for a normal app.

Garbage collection shows up in the tail

Go has a garbage collector. It is fast, and most apps never feel it. But it still has to do work to free memory, and that work can land inside the millisecond timings I report. Run tens of thousands of checks at once and a pause at the wrong moment lifts a p99 number. At that point I am measuring my own runtime, not your server.

Rust has no garbage collector. Memory is freed at a point I can see in the code. There is no background pause I did not write. For a tool that sells timing, that control is worth the extra work Rust asks for.

What that looks like in memory

Here are some real numbers, with a warning attached. They come from a load test on a developer laptop, not a production server. I use them to catch a slowdown between two versions of my code, not to plan capacity. A real server does better, a small box does worse. Treat them as a floor.

In one run, a single machine held 50,000 checks in flight and peaked at 933 MiB of memory. That is under one gigabyte for fifty thousand live checks. My running server uses about 42 MiB while watching its monitors, and it sits quiet when there is nothing to do. That kind of density is normal for a service with no garbage collector, and it means one small box covers a lot of monitors. I go deeper on the prober and the throughput numbers in the build story.

The bug that never compiles

Speed is only half of it. The other half is a class of bug that Rust refuses to build.

Picture many workers writing to one shared map of results at the same time. In Go this compiles and runs. Sometimes it is fine. Sometimes two goroutines write at once, you get corrupt data or a crash, and it only happens under load, which is the worst time to find out:

// compiles fine, races at runtime
results := map[string]int{}
for _, check := range checks {
    go func(c Check) {
        results[c.ID] = c.Latency // data race
    }(check)
}

Go gives you good tools for this. There is a race detector, a sync.Mutex, and channels. But remembering to reach for them is on you.

In Rust the same concurrent write does not compile. The compiler stops you until the shared map is wrapped in a lock:

// will not compile unless the shared map is a Mutex
let results = Mutex::new(HashMap::new());
std::thread::scope(|s| {
    for check in checks {
        let results = &results;
        s.spawn(move || {
            // each worker locks only for its own insert
            results.lock().unwrap().insert(check.id, check.latency);
        });
    }
});

Each worker holds the lock only for its own insert, so the writes stay safe without blocking the others for long. For code that runs day and night across many machines, "the compiler will not let you ship the race" removes a whole set of late-night bugs before they exist.

Where Go is the better pick

None of this makes Go a bad choice. Often it is the better one. Go is faster to learn. It builds in seconds. Its standard library for network services is excellent, and a new engineer can be useful in days. If I were building a normal web service, or something I had to ship this week, Go would be on the table and might win.

Rust asks more from you first. The compiler argues with you. The build is slower. You spend time on things Go would just handle. I take that trade because this job is narrow and it rewards tight control over memory and timing.

The compiler as a safety net

Heavy code review is how large teams catch data races and memory bugs. Rust gives you a lot of that for free. The compiler turns those mistakes into build errors, so they never reach production and never wake anyone up. Every change gets a strong, automatic check before it ships, which is a big part of why I trust the service to run unattended.

It is also why Uptimepage is open source and self-hostable. You can read the code, run it on your own hardware, and export your data whenever you want, so you are never locked into a single vendor. You can start from GitHub.

The honest version

The honest version is not "Rust beats Go." It is "for a tool that lives or dies by timing and runs at high concurrency, Rust fit better." Pick the language for the job in front of you. Mine happened to be a job that Rust is very good at.

If you have shipped a heavily concurrent network service, did you reach for Go or Rust, and did the GC ever actually show up in your tail latencies? Curious where people land when the timing is the product.

Is 98% uptime good? It allows 7.3 days of downtime a year

Slim — Fri, 17 Jul 2026 13:26:37 +0000

Cover photo by the blowup on Unsplash.

I build an uptime monitor, so this question lands in my inbox a lot: is 98% uptime good?

For a public website or a paid API, no. For an internal tool or a side project, it is fine. The difference is one division away.

98% looks like a top grade because school taught us that 98 out of 100 is excellent. Uptime does not grade like school. The whole scale for public services lives between 99% and 100%, and serious targets differ only in the digits after the decimal. On that scale, 98% sits at the bottom.

Turn the percentage into time

The allowed failure at 98% is 2%. Two percent of a year is 7.3 days. Two percent of a 30-day month is 14.4 hours. If your shop makes $2,000 a day, 98% uptime means you accept about $14,600 of closed-door time per year and the target still counts as met.

Here is the same math for every common target. The last column prices the downtime for that $2,000-a-day shop.

Uptime	Per day	Per 30-day month	Per year	Lost sales per year
90%	2.4 hours	3 days	36.5 days	$73,000
95%	1.2 hours	36 hours	18.3 days	$36,500
98%	28.8 minutes	14.4 hours	7.3 days	$14,600
99%	14.4 minutes	7.2 hours	3.7 days	$7,300
99.5%	7.2 minutes	3.6 hours	1.8 days	$3,650
99.9%	1.4 minutes	43 minutes	8.8 hours	$730
99.95%	43 seconds	21.6 minutes	4.4 hours	$365
99.99%	8.6 seconds	4.3 minutes	52.6 minutes	$73
99.999%	0.9 seconds	26 seconds	5.3 minutes	$7

Two jumps on this table do most of the work in real contracts. From 98% to 99.9%, the allowed downtime drops from 14.4 hours a month to 43 minutes. From 99.9% to 99.99%, it drops from 43 minutes to 4.3 minutes, and that second jump usually costs about ten times more engineering than the first while saving the shop $657 a year.

The shape of the downtime matters

98% per month is 14.4 hours, but the number says nothing about how those hours land.

Thirty minutes of planned maintenance every night at 03:00 adds up to 98% and most users never notice. One 14-hour outage on the day of your product launch is also 98%. Same score, very different month.

So a single uptime percentage is a summary, not the full story. When someone quotes you a number, also ask about the longest single outage and when it happened.

When 98% is enough

Plenty of systems can live at 98% and nobody gets hurt:

An internal wiki. People retry after lunch.
A staging environment. Downtime there is often planned.
A batch job that builds reports at night. It has hours of slack before anyone reads the output.
A home server on a residential connection. Your power company already decided your uptime for you.

The shared pattern: when these go down, nobody loses money and nobody loses trust. Paying for more nines there is waste.

When it is not

A checkout page, a paid API, a login service. Here 98% fails twice. First the direct cost: 14.4 hours a month of failed requests and support tickets. Second the trust cost, which is larger and slower. A customer who hits your outage twice in one week does not check your uptime report. They remember that your service is the one that breaks.

One more trap: an SLA is not uptime. Uptime is the measured number. An SLA is a contract promise with a penalty, and the penalty is almost always a service credit. If a provider misses its 99.9% SLA, you get part of your bill back. Your customers get nothing.

What to aim for instead

99.9% is the default target for customer-facing services for a reason. 43 minutes a month is enough room for a bad deploy and a couple of small failures, and a small team can hit it without heroics. Above that, each nine costs roughly ten times more and most users cannot feel the difference.

If you want to run your own numbers, I keep a free uptime SLA calculator and an error budget calculator on our site; both work without signup.

What target do you actually run in production, and did you pick it or inherit it? I am curious how many teams measured before they promised.

The status page you can't fake: measured uptime, not published

Slim — Wed, 15 Jul 2026 13:50:26 +0000

A status page is the one dashboard a company publishes about its own service. It is also the one place where the company has a reason to look good. That is a problem. If the page can be edited to look better than reality, it stops being useful. So when you build or choose a status page, ask one thing: can someone hide a real outage on it?

The short answer: a status page you can trust builds its uptime bar from real checks, not from the incidents a person chose to publish.

Two layers, and only one is yours to edit

A good status page has two parts. The first part is the measured data: the green and red timeline, the 90-day bar, the uptime percent. The second part is incidents: the notes a person writes to explain what broke, what they are doing, and when it is fixed. On the page they sit next to each other and look the same. They are not the same, and mixing them is where trust gets lost.

The measured data answers one question: what did the checks see? The incident notes answer a different one: what does the team want to say about it? The first is a fact. The second is a story. A status page you can trust lets the team write the story, but keeps them away from the facts.

The story layer also includes the postmortem, the write-up you post after an outage. A postmortem is honesty you add on purpose. You explain what broke and why, because you choose to. The bar works the other way. It shows the failure on its own, whether you write anything or not. You control the story. You do not control the facts.

The trap: a bar that only shows what you published

The common mistake is to build the uptime bar from incidents. It feels natural. You already open an incident when something breaks, so why not color the timeline from incidents too? Now the bar turns red only where an incident exists and is marked public.

The problem comes on the day an outage has no public incident. Maybe no one published it. Maybe a setting was wrong. Maybe the monitor was added to the page after the outage, so its incident was saved as private and never checked again. In every case the timeline shows green over a real red day, and it does this quietly. The uptime number goes up. The customer sees 100 percent over a week they remember as broken.

One version of this trap is easy to build by accident, so it is worth explaining. You decide "is this incident public?" once, at the moment the incident opens, based on whether the monitor was on a public page right then. Then you save that answer and never look again. In the code it looks like a live check. It is really a photo taken one time. Move the monitor onto the page a day later, and its past outages stay hidden, because the photo was taken before the monitor was there. Any yes or no mark that is set once and then trusted forever has this problem.

The fix is a rule, not a switch

The bar should come from the measured data. It needs only one rule to stay calm: wait for confirmation before you count downtime. You do not want one failed check from one place to turn a whole day red, because networks are noisy and one bad checker is not an outage. So you wait for agreement: more than one region failing at the same time, for more than one check in a row. That one rule is enough to keep a short blip from becoming a red day.

One thing matters here: the same rule feeds both your alerts and your uptime bar. If the rule that wakes your on-call person is the same rule that colors the timeline, the two can never tell different stories. The moment you add a second rule just for the bar, it will drift from the first, and the page will disagree with itself. On-call gets paged, but the public history says everything was fine.

Publishing stays where it belongs, on the notes. The team decides whether to write an incident, what to say, and when to post the all-clear. They do not decide whether last Tuesday was down. The checks already decided that.

Test the status page you already have

You can check any status page, including your own, in a few minutes.

Add a monitor to a page after it has already had an outage. Does the history show the outage, or does it start clean from the day you added the monitor?
Take a real incident and unpublish it. Does the bar keep the red day, or does the day turn green?
Make one region fail for one second. Does the whole day go red, or does the bar stay calm?

A page you can trust shows the outage in the first test, keeps the red in the second, and stays calm in the third. A page that fails these is usually not lying on purpose. It just built its timeline on top of what people chose to publish, and that always has holes.

Where this leaves you

This is how I built it into Uptimepage: the 90-day bar and each status light come from confirmed downtime, measured across regions with a confirmation rule, not from what someone chose to publish. Incidents and postmortems are the layer you write by hand. It is AGPL and open source on GitHub, and there is a longer write-up on the internals if you want the Rust side.

You should not be able to fake your uptime. You should not be able to fake it by accident either. The bar is a measurement. Keep it one.

How does your status page compute its uptime bar, from checks or from published incidents? Curious what people run and whether anyone has been bitten by the frozen-flag version of this.

Error budgets, explained: SLOs, burn rate, and when to stop shipping

Slim — Mon, 13 Jul 2026 14:11:33 +0000

The idea

An uptime target has a second number hidden inside it. Say you promise 99.9% uptime. You are also saying that 0.1% is allowed to fail. Put that 0.1% into real time, and that is your error budget: the downtime you can have before you break the promise.

This changes how you look at downtime. It stops being a mistake to feel bad about and becomes a budget you can spend: on a risky deploy, on a slow service you depend on, or on a migration. When the budget runs low you slow down. When it is healthy you can move fast.

The formula

An SLO is your reliability target, for example 99.9%. The gap to 100% is the failure you are allowed. Multiply that gap by the length of the window, and you get real time you can spend.

error budget = window x (1 - SLO)

A 99.9% SLO over a 30-day month is 2,592,000 seconds times 0.001. That is 43 minutes and 12 seconds. That is the whole budget for the month, shared across every incident, not a fresh 43 minutes each day.

So three nines is not "never go down". It is a 43-minute budget each month. Every extra nine costs about ten times more engineering, for downtime that most users never notice. This is why Google says that 100% is the wrong target for almost every service.

Burn rate is the speed

The total tells you how much you can spend. It does not tell you how fast. Two services can both sit at 99.9% for the month: one loses the budget slowly, the other loses it all in a single bad hour. Burn rate tells them apart.

burn rate = (1 - measured) / (1 - SLO)

A burn rate of 1x spends the whole window exactly. 2x spends it in half the time. Below 1x, you finish the month with budget left. Above 1x, the rate tells you the deadline:

budget runs out in = window / burn rate

At 2x, a 30-day budget is gone in fifteen days. At 14.4x it is gone in about two days. That same 14.4x spends 2% of the budget in one hour, which is the level most fast alerts are set to.

Turning burn rate into alerts

One threshold is not enough. It either alerts too late or it sends too many false alarms. The common fix uses two windows: a long one to confirm the problem is real, and a short one to clear the alert quickly once you fix it. Both have to be burning for the alert to fire. For a 30-day budget, these are the usual settings:

Fast page: 2% of the budget in 1 hour (with a 5-minute short window). This is a 14.4x burn.
Page: 5% in 6 hours (30-minute short window). This is a 6x burn.
Slow ticket: 10% in 3 days (6-hour short window). This is a 1x burn.

The fast page catches a sudden outage. The slow ticket catches a slow problem that would still use up the whole month if nobody looked.

The rule is the point

The math is the easy part. The value comes from a rule you agree on before anything breaks. The rule is simple. When the budget runs out, risky launches stop, and the team works on reliability until the budget grows back. While the budget is healthy, you ship and you take the risk.

One more rule keeps it fair. If you never spend your budget, your SLO is too strict, and you are paying for reliability that no user asked for.

Try it

I put the formulas into a free, no sign-up error budget calculator: enter an SLO and your measured availability, and it shows budget spent, budget left, burn rate, and a burn-down chart. There is also an uptime SLA calculator for the full downtime-per-nine table.

What SLO and burn-rate thresholds do you run in production? Curious how others pick them.

ClickHouse system tables ate my disk (and the fix)

Slim — Thu, 09 Jul 2026 15:13:13 +0000

TL;DR.

A monitoring alert said I was dropping check results. The disk was 100% full. My actual data was 20 MB. ClickHouse had quietly written 12 GB of logs about itself, mostly system.text_log and system.trace_log. Those same logs also burn CPU at idle. The fix is a few lines of ClickHouse config that disable the noisy logs and slow the metrics collector. Full config is below.

The alert

One morning a critical alert fired: UptimepageResultsLost. Its description is blunt: storage write failures or dropped results, checks run but results are not persisting. Then it did something worse than fire once. It cleared, fired again, cleared, and fired again, over and over.

I run Uptimepage, an uptime monitoring service. "Results not persisting" means the one thing customers pay for, recording whether their sites are up, might be failing. So it had my full attention.

The good news first: no data was lost. The write path retries, and every failed write was caught by a retry. I keep a counter for results that actually get dropped, and it stayed at zero the whole time. But something was clearly wrong underneath.

The real signal: a full disk

The first real signal came from Postgres, which had crashed and restarted a couple of minutes earlier:

FATAL: could not write lock file "postmaster.pid": No space left on device

The disk was 100% full. Postgres could not write its lock file, crashed, and recovered on its own through WAL replay. ClickHouse, where I store raw check results, was rejecting inserts with its own version of the same complaint:

Code: 243. DB::Exception: Cannot reserve 1.00 MiB, not enough space:
While executing WaitForAsyncInsert. (NOT_ENOUGH_SPACE)

So the alert was a symptom. The real problem was a full disk. That reframes the question: I do not store much, so what filled it?

Leak one: old Docker images

docker system df gave the first half of the answer:

88 Docker images, only two of them in use. Every deploy pulls a fresh image, and nothing pruned the old ones, so they piled up for weeks. A docker image prune -af reclaimed about 10 GB and took the disk off the ceiling.

That stopped the bleeding. But 13.8 GB of volumes is a lot for a service whose data I thought was tiny. That number turned out to be the real story.

Leak two: ClickHouse logging about itself

I went into ClickHouse and asked the obvious question, which table is big:

SELECT database, formatReadableSize(sum(bytes_on_disk)) AS size, sum(rows) AS rows
FROM system.parts
WHERE active
GROUP BY database
ORDER BY sum(bytes_on_disk) DESC

The answer stopped me:

Database	Size	Rows
`system`	11.8 GiB	577,340,927
`monitor`	19.7 MiB	1,813,884

monitor is my data: every check result and rollup I keep. About 20 MB. The system database, ClickHouse's own diagnostic tables, was 11.8 GiB. Nearly all of the storage was ClickHouse logging about itself.

Breaking system down by table:

Two tables did most of the damage. system.text_log (5.29 GiB) is a copy of the server's own log output written into a table. system.trace_log (3.02 GiB) is the query profiler, which samples running queries. Both are handy when you are actively debugging ClickHouse. Neither is worth multiple gigabytes when I am not. And text_log is off by default, so something in my setup had switched it on.

The fix

Two parts: reclaim the space now, and stop it coming back.

Reclaim now. The system log tables are throwaway diagnostics, not real data. Truncate them:

TRUNCATE TABLE IF EXISTS system.text_log;
TRUNCATE TABLE IF EXISTS system.trace_log;
-- and the rest of the system.*_log tables

That gave back 12 GB at once and took the disk from 74% down to 41%.

Stop the regrowth. Truncating is a one-time cleanup. Without a cap, the logs fill right back up. The durable fix is ClickHouse config, added under /etc/clickhouse-server/config.d/. I watch ClickHouse through Grafana, not these tables. So I disable almost all of them and keep only query_log and part_log, both bounded, for the rare hands-on debugging session:

<clickhouse>
    <!-- Sample async metrics every 60s, not every second. -->
    <asynchronous_metrics_update_period_s>60</asynchronous_metrics_update_period_s>

    <!-- Disable the log tables. remove="1" on an absent table is a no-op,
         so this list is safe to paste as-is across ClickHouse versions. -->
    <asynchronous_metric_log remove="1"/>
    <asynchronous_insert_log remove="1"/>
    <backup_log remove="1"/>
    <error_log remove="1"/>
    <crash_log remove="1"/>
    <metric_log remove="1"/>
    <query_metric_log remove="1"/>
    <query_thread_log remove="1"/>
    <query_views_log remove="1"/>
    <session_log remove="1"/>
    <text_log remove="1"/>
    <trace_log remove="1"/>
    <opentelemetry_span_log remove="1"/>
    <zookeeper_log remove="1"/>
    <processors_profile_log remove="1"/>
    <latency_log remove="1"/>
    <background_schedule_pool_log remove="1"/>

    <!-- Keep query_log and part_log, bounded, for on-hand debugging. -->
    <query_log>
        <ttl>event_date + INTERVAL 3 DAY DELETE</ttl>
        <max_size_rows>1048576</max_size_rows>
    </query_log>
    <part_log>
        <ttl>event_date + INTERVAL 3 DAY DELETE</ttl>
        <max_size_rows>1048576</max_size_rows>
    </part_log>
</clickhouse>

remove="1" disables a system log completely. The <ttl> element on query_log and part_log adds a TTL and keeps the table's default partitioning, so you do not have to restate the whole engine. ClickHouse picks up the change after a restart. Altinity's "system tables ate my disk" note covers the same ground and is worth a read. If you would rather keep the diagnostics, give every log a short <ttl> instead of disabling it.

Bonus: lower CPU, not just disk

Those log tables are written constantly, and on a small server that steady write load shows up as CPU. A long-running ClickHouse issue tracks the server using noticeable CPU at zero load, #60016. People there report dropping from 40 to 70% CPU down to about 1.5% after disabling the logs and slowing the async-metrics collector. So those two settings, the asynchronous_metrics_update_period_s line and the remove="1" block, pay off twice: less disk and less CPU.

The real lesson: alert on the cause, not the effect

Here is the part that stings. I had an alert for "results are being dropped." I did not have an alert for "the disk is filling up." So a full disk is a slow, predictable problem that builds over days. But it only reached me as a sudden downstream symptom, after Postgres had already crashed once.

A downstream alert like "results lost" is not a substitute for watching the resource that actually runs out. I added the missing one: a plain host disk-space alert on node_filesystem_avail_bytes, firing at 80% and 90% used, well before anything starts failing. That is the alert that should have caught this on day one.

Key takeaways

ClickHouse system log tables are unbounded by default and can dwarf your real data. Mine were 11.8 GB against 20 MB of actual data.

system.text_log and system.trace_log are the usual offenders. text_log is off by default, so check whether something enabled it.

Cap them in config: remove="1" to disable a log, or <ttl> plus <max_size_rows> to bound the ones you keep. Truncate to reclaim space right away.

It is not just disk. The same logs burn CPU at idle on small servers. Disabling them, plus asynchronous_metrics_update_period_s = 60, took reporters in ClickHouse issue #60016 from 40 to 70% CPU down to about 1.5%.

Anything that pulls artifacts on a schedule, Docker images in my case, needs matching cleanup or it becomes a slow disk leak.

Alert on the cause (disk space), not only the effect (dropped writes). The cause gives you days of warning; the effect gives you minutes.

This happened on Uptimepage, the uptime monitor I run and dogfood.

Postgres vs ClickHouse? I use both. 4 tricks from the split.

Slim — Thu, 09 Jul 2026 05:55:03 +0000

TL;DR. My uptime monitor keeps config and incidents in Postgres and check results in ClickHouse. The split is one rule: does a row ever change? Config gets edited, so it wants Postgres transactions and constraints. A check result is written once and never touched again, so it goes to ClickHouse, where the right codec, a careful sort key, and a per-row TTL make billions of rows cheap. Four tricks and a bonus below, useful even if you only ever run one database.

Every monitor I run writes a row every 20 seconds, from every region, and never stops. One monitor is about 4,300 rows a day, per region. Multiply by every monitor on the platform and the rows only ever go up.

That stream would slowly crush a normal Postgres table, and watching it is the whole product. So the check results do not live in Postgres. They live in ClickHouse. Everything else, the monitors and incidents and teams, lives in Postgres. The interesting part is the line between them.

People ask why not one database. "Postgres vs ClickHouse" is the wrong question, because the two are not fighting over the same job. Here is the line I draw, and the one rule under all of it: split your data by how it is written, not by which engine is faster.

1. Does the row change? That picks the database

Forget benchmarks for a second. One question sorts a table into one store or the other: after you write a row, will you ever change it?

one monitor, two stores:

  you edit a monitor  ->  Postgres   (targets, incidents, team)
                          the row changes, has constraints, lives in a transaction

  a probe checks it   ->  ClickHouse (check_results)
                          one row, written once, never updated

A monitor is a row that changes. You toggle it off, edit the URL, change the interval. So it lives in Postgres:

CREATE TABLE targets (
    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name          TEXT NOT NULL,
    check_spec    JSONB NOT NULL,
    interval_secs INTEGER NOT NULL CHECK (interval_secs >= 10),
    enabled       BOOLEAN NOT NULL DEFAULT true,
    updated_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

That CHECK means a 3-second interval can never reach the table, no matter which part of the app tried to write it. This is Postgres doing the thing it is best at: a small set of rows that must stay correct while many callers change them at once.

A check result is the opposite. It is written once when a probe finishes, and then it never changes. Nothing ever updates it, so it does not need any of that:

CREATE TABLE check_results (
    org_id      UUID,
    target_id   UUID,
    region      LowCardinality(String),
    timestamp   DateTime('UTC') CODEC(DoubleDelta, ZSTD(1)),
    status      Enum8('up' = 1, 'down' = 2, 'degraded' = 3, 'error' = 4),
    duration_ms UInt32 CODEC(T64, ZSTD(1))
) ENGINE = MergeTree

No updated_at, no foreign key, no transaction. Just an append-only stream that grows forever. That is the shape ClickHouse is built for and the shape that slowly hurts Postgres.

Notice the types too. status is an Enum8, one byte on disk, not the string "up". region is LowCardinality, stored once in a dictionary and referenced by a small id instead of repeating the text on every row. On a table that only ever grows, a byte saved per row is a byte saved times billions.

2. The right codec turns a day of timestamps into almost nothing

Once the results are in a column store, the type on each column is most of the compression, and the default setting wastes a lot of space.

Look at that timestamp again. A monitor checks every 20 seconds, so for one monitor a day of timestamps is a run of 20, 20, 20, 20, over and over. DoubleDelta stores the change in the change. For a steady interval the first delta is a constant 20 and the second delta is zero, so each row after the first packs down to about a bit. The codec is not a ClickHouse invention: it comes from Facebook's Gorilla time-series paper, built for exactly this, measurements taken at a steady rate.

timestamp   DateTime('UTC')  CODEC(DoubleDelta, ZSTD(1)),
duration_ms UInt32           CODEC(T64, ZSTD(1)),
dns_ms      Nullable(UInt16) CODEC(T64, ZSTD(1)),

I also store the timestamp at whole seconds on purpose. The smallest interval is 20 seconds, so no two checks for one monitor ever land in the same second, and the sub-second detail lives in duration_ms where it belongs. Whole seconds that step by a fixed amount compress far harder than milliseconds would.

The latency columns get T64 instead, because they are small integers that stay in a small range, a different shape from a steady clock. T64 crops the unused high bits off a block of values, so a UInt32 that never climbs past a few thousand milliseconds stops paying for all 32 bits. That is the trick worth copying, and it is not the specific codec names. It is that a timestamp ticking by a fixed step and a latency staying in a small range are two different shapes, and telling the database which is which does the work.

3. Put the tenant in the sort key, not the partition

I got this one wrong first, so learn it from my mistake instead of your own.

This is a multi-tenant app, so every row carries an org_id and almost every query filters by it. My first schema partitioned by org, one slot per customer, because it felt tidy. ClickHouse turned that into a flood of tiny parts, the merges could not keep up, and startup got slower the more customers I had. I ripped it out. A partition is not a folder for tidiness. It is a physical unit ClickHouse merges and expires, and you want few large ones, not many small ones.

Here is what it should be:

) ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (org_id, target_id, region, timestamp)

org_id leads the ORDER BY, so a per-org query walks the sorted primary index and reads only that org's slice. But it stays out of PARTITION BY. The rule I follow now: partition by something low-cardinality that you also delete by, here the day, and put the high-cardinality tenant key in the sort order. Sort key answers "find this org fast". Partition answers "drop old data cheaply", which is the next trick.

4. Make retention a number in a row, not a schema change

Different plans keep history for different lengths of time. The clumsy way is a migration or a cleanup job per plan. The clean way is to make the retention window a column:

ttl_days UInt16 DEFAULT 30 CODEC(ZSTD(1))
) ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (org_id, target_id, region, timestamp)
TTL timestamp + toIntervalDay(ttl_days)

Each row carries its own ttl_days, stamped from the org's plan at the moment it is written. A free plan keeps 30 days, a paid plan keeps more, and changing that needs no ALTER, no migration, no backfill. The next write just stores a different number. The daily partitions line up with the TTL, so ClickHouse expires old data by dropping whole parts, close to free, and a whole column of the same 30 compresses away to nothing. The flexibility costs no space.

One more piece makes reading that history cheap. A materialized view rolls raw checks into per-minute and per-hour summaries as they land:

CREATE MATERIALIZED VIEW check_results_1m
ENGINE = AggregatingMergeTree
ORDER BY (org_id, target_id, region, minute)
AS SELECT
    org_id, target_id, region,
    toStartOfMinute(timestamp)  AS minute,
    countState()                AS total_checks,
    countIfState(status = 'up') AS up_checks,
    avgState(duration_ms)       AS avg_duration_ms
FROM check_results
GROUP BY org_id, target_id, region, minute;

So a dashboard showing 30 days across a thousand monitors reads a few thousand minute-buckets, not millions of raw rows. Recent views read the minute rollup, long history reads an hour rollup, and the raw rows underneath expire on their own TTL. The firehose is there when you need to drill into one bad minute, and left alone the rest of the time.

One more, because everyone hits it: make inserts idempotent

Here is the trick I wish someone had told me first. My agents batch check results and send them to ClickHouse. A network blip after the server commits but before my side gets the ack means the batch retries and sends the exact same block again. Without protection, that double-counts every row in it, and your uptime numbers quietly drift.

ClickHouse has a fix built in, but for a plain (non-Replicated) MergeTree it is off until you turn it on:

SETTINGS index_granularity = 8192,
         non_replicated_deduplication_window = 1000;

The server hashes each inserted block and remembers the last 1,000 hashes. A retry sends an identical block, the hash matches, and the server drops it instead of appending it again. The retry becomes safe to do blindly, so the client stays simple: send, and if unsure, send again. Make the window bigger than the most blocks a single retry could resend, and you stop having to worry about duplicate writes.

So, one database or two?

For almost every app, one. If your data fits in Postgres and stays fast, a second database is a cost you should not pay: two schemas, two clients, two things to back up and think about.

Reach for the second store only when one table stops looking like the rest of your tables. For me that table was check_results. It is written once and never edited, it grows without end, and every question I ask it is a summary over a time range. That is a different shape from my monitors and my incidents, so it wanted a different database.

The rule I would give my past self: do not split by "which database is faster". Split by how the data is written. Rows that change and must stay correct want Postgres. An append-only stream you only ever summarize wants a column store like ClickHouse. Most of the tricks above are that one idea pushed down into the schema.

Uptimepage is open source, AGPL-3.0, and both schemas are in the repo: github.com/uptimepage/uptimepage. The probe that writes those rows is its own post, and the wider build story, one binary and two databases, is here. Or start free on the hosted tier and point a check at something.

4 things writing an HTTP prober in Rust taught me

Slim — Tue, 07 Jul 2026 14:07:03 +0000

TL;DR. An uptime prober is a strange HTTP client, so I swapped reqwest for raw hyper. It needs the opposite of what a normal client gives you: no connection pool, so the cold path gets measured instead of hidden; every phase timed, even when connect fails; a hard SSRF block on the URL the user typed; and exact failure reasons instead of one flat error. Four tricks below, each worth copying even if you never build a monitor.

I built an uptime checker in Rust. Like most people, I used reqwest first. Then I dropped it and moved down to raw hyper.

Not because reqwest is slow. It is great, and for app code you should use it. But a prober is a strange kind of HTTP client. It makes one request, and its whole job is to measure that request and report exactly what happened. A normal client is built to do the opposite: make requests fast and hide the messy details. Every trick below is a messy detail I needed to keep.

Here are four things building it taught me. Each one is a habit worth copying, even if you never write a monitor.

1. A connection pool lies to a monitor

The first thing reqwest gives you is a connection pool, and it is the first thing I had to remove.

A pool keeps TCP connections open and reuses them. The second request to a host skips DNS, skips the TCP handshake, skips TLS, and comes back much faster. For app code that is a free speed boost. For a monitor it is a lie. If I reuse a warm connection, the "connect time" I report is near zero, because I did not connect, I borrowed. A user watching a slow TLS handshake slowly rise in one region would see nothing, because the pool hid the handshake.

So the connector makes exactly one request per connection, then drops it:

/// No pooling: the caller drives exactly one request over the returned
/// stream, then drops it.
pub(crate) async fn timed_connect(
    p: &ConnectParams<'_>,
    host: &str,
    port: u16,
    is_https: bool,
) -> Result<TimedConnection, ConnectError> {

Every check pays the full cost of a fresh connection, because the full cost is the thing I am trying to measure. The lesson is general: if you are timing a request, a warm pool is measuring the wrong thing. Reuse is a feature that erases exactly the numbers a monitor exists to show.

2. Put the timings inside the error

Splitting a request into DNS, TCP, TLS, and time to first byte is easy on the happy path. The interesting question is what happens when connect fails.

The naive design loses everything on failure. You get an Err and no numbers, so a hung TLS handshake looks the same as a dead DNS server. Both are just "failed." That is the worst moment to have no data.

The fix is small, and I now use it everywhere: make the error carry the phases that finished before the one that broke.

pub(crate) enum ConnectError {
    Dns(anyhow::Error),
    NoAddrs { dns_ms: u16 },
    Connect { err: io::Error, dns_ms: u16 },
    Tls { err: io::Error, dns_ms: u16, connect_ms: u16 },
}

A TLS failure still reports how long DNS and TCP took. So when a cert handshake hangs, the chart still shows DNS was 12ms, connect was 45ms, and then TLS took the rest. The shape of the error tells you what broke. You are never left guessing which layer stalled, even on the path where everything went wrong. reqwest gives you one flat Error at the very end. Here the error type holds the answer.

one probe, one fresh connection, each phase on its own clock:

  DNS         TCP connect      TLS handshake     request + TTFB
  +-----------+----------------+-----------------+---------------->
   12ms        45ms             (hung)            never reached

  TLS failed. The error still carries dns_ms=12, connect_ms=45,
  so you know exactly which layer stalled, not just "it failed".

3. The SSRF check goes after DNS, not before

This is the one most people building a fetch-any-URL feature get wrong, and it is the scary one.

A checker fetches whatever URL the user typed. That is the feature. It is also a classic SSRF engine. Someone signs up, points a check at http://169.254.169.254/latest/meta-data/, and now your prober is reading cloud credentials from inside your own network and showing them the reply. Point it at http://10.0.0.5:6379 and it is a port scanner for your private subnet.

The trap is thinking you can block this by checking the hostname. You cannot. The classic bypass is DNS rebinding: the name looks public and passes your check, but it resolves to 127.0.0.1. The block has to happen after resolution, on the real IP you are about to dial:

let addrs: Vec<SocketAddr> = p
    .resolver
    .resolve_addrs(host)
    .await
    .map_err(ConnectError::Dns)?
    .into_iter()
    .filter(|ip| p.ssrf_guard.allow(*ip))   // block loopback, private, link-local, metadata
    .map(|ip| SocketAddr::new(ip, port))
    .collect();

Resolve first, filter every resolved IP, then connect only to what survives. And because each redirect hop reconnects through this same guarded path, a Location: header pointing at 169.254.169.254 gets rejected exactly like a directly typed one. The safety sits in the single place all connections pass through, so there is nothing to forget on the redirect path. If you ever fetch a user-supplied URL server-side, this is the check that matters, and the hostname version is not it.

4. Stop grepping your error strings

When a check goes red the user gets one line, and that line has to be right. "Something went wrong" trains people to ignore alerts. "Certificate expired" tells them what to fix in five seconds.

The tempting way to produce that line is to match on error message text. It is a trap: those strings change between library versions and read differently on every OS. The message is not an API. So instead the connector downcasts to the real error and reads the reason straight out of it:

fn tls_reason(io: &io::Error) -> &'static str {
    use rustls::CertificateError;
    let Some(rustls::Error::InvalidCertificate(cert)) =
        io.get_ref().and_then(|e| e.downcast_ref::<rustls::Error>())
    else {
        return "tls";
    };
    match cert {
        CertificateError::Expired => "certificate expired",
        CertificateError::UnknownIssuer => "certificate not trusted",
        CertificateError::NotValidForName => "certificate hostname mismatch",
        CertificateError::Revoked => "certificate revoked",
        _ => "certificate invalid",
    }
}

An expired cert, an untrusted issuer, and a hostname mismatch are three different problems with three different fixes, and the user should never have to guess which one they have. The io::ErrorKind and the rustls error already carry the truth. The trick is to refuse to flatten it into a string and then re-parse the string you just made.

So, reqwest or not?

For your program? Almost certainly reqwest. Going down to hyper means you own redirects, decompression with a size cap, timeout budgeting per hop, and the request-target rules for HTTP/1 versus HTTP/2. That is real code with real edge cases, and every one is a place reqwest would have been correct for free. If I needed to fetch a config file at startup, writing this connector would be a mistake.

It is worth it here for one reason: measuring and guarding the request is the product, not a detail of it. When the request itself is the thing you sell, you want to own every millisecond and every IP it touches.

Uptimepage is open source, AGPL-3.0, and the whole probe path is in the repo: github.com/uptimepage/uptimepage. If you want the wider build story, one binary and two databases, that is a separate post. Or just start free on the hosted tier and point a check at something.

Email bombing through uptime monitoring pages

Slim — Mon, 06 Jul 2026 13:14:34 +0000

Cover photo by Brian J. Tromp on Unsplash.

I build an uptime monitoring tool. Part of my job is to attack it before someone else does. This post is about one attack that most people in this space do not talk about: using monitoring tools to flood someone's inbox with email.

I will show you how it works from the attacker side. Then I will show you how to stop it from the builder side. If you run any tool that sends email to an address a user typed in, this is your problem too.

What email bombing is

Email bombing is simple. You flood one inbox with so many messages that the person cannot use it. Hundreds of emails in a few minutes. When the attacker uses signup forms to do it, people call it subscription bombing or list bombing. Same idea. Anti-abuse groups describe it as a denial-of-service attack on your inbox, and IT teams saw a rise in 2026.

The goal is usually not the flood itself. The flood is a cover.

Say an attacker just stole your card and bought something. Your bank sends an email or a text: "Did you make this payment?" If your inbox has hundreds of new emails in a few minutes, you will never see that one bank message. By the time you dig it out, the money is gone. The flood buys the attacker time and quiet. Security teams have watched attackers use this trick to hide real fraud.

It is not always about money. Some ransomware crews use email bombing as step one. They flood a worker with mail, then call and pretend to be the IT help desk: "We see the spam, let us help you fix it." The panic is the way in. The flood softens the target for the real attack. The US cyber agency CISA documented this play: email bombing plus a fake help-desk chat on Microsoft Teams, used by the Black Basta ransomware crew to get in. Former members kept using it through 2025.

Why uptime pages make a good weapon

Photo by Moritz Erken on Unsplash.

Here is the part that surprised me the first time I looked.

Most uptime and status tools let you add an email address for alerts. You type an address, the tool sends a "confirm your email" message, and you click a link to turn alerts on. Normal stuff.

Now look at it as an attacker. The tool will send an email to any address I type. I do not need to own that address. I type the victim's email and hit save. The confirm email goes to the victim, not to me.

One email is not a bomb. But there are many of these tools. And inside each tool I can make many monitors, each with its own alert email. So I write a small script. Sign up, add the victim's address to fifty monitors, move to the next tool, repeat. The victim gets a wave of "please confirm" emails from services they have never heard of.

The mean part is where these emails come from. They come from real companies with a good name. They pass the checks that tell Gmail a sender is real (SPF and DKIM, if you want the terms). So they land in the main inbox, not in spam. One study in 2026 found that most of these emails were not marked as spam and reached the inbox. A normal spam filter usually will not save the victim, because on paper none of it is spam.

The builder side: how not to be the tool that floods people

I will be honest. When I first tested my own tool, it had this hole. The confirm email had no real limit and no clear owner. So I fixed it. Here is what actually helped, in order.

Rate limits are the floor. Cap how many confirm emails one address can get per day across your whole platform, not just per account. If one attacker opens twenty accounts, a per-account limit does nothing. Count by the target address.

Say who and why. My old confirm email just said "confirm this address." Now it says who added it and to what: "Acme Inc added this address to a monitor." If a stranger gets one, at least they can see what is happening.

Give a fast way out. Put a one-click "this was not me, stop" link in the email. One click kills that address on your tool. No login, no support ticket. There is a standard for this, RFC 8058, so mail apps can show a stop button right in the message.

Block the obvious traps. Do not allow role addresses like postmaster@ or abuse@. They are shared mailboxes, not a person who asked for alerts. And do not allow your own company domains, because a tool that emails itself can start a mail loop.

Skip the email when you can. If the person adding the address is logged in with that same email through Google or GitHub, you already know they own it. Mark it confirmed and send nothing.

None of this is clever. It is boring plumbing. But boring plumbing is what keeps your name off the list of services used in someone's email bomb.

If you are the one getting bombed

Do not delete everything in a panic. That is exactly what the attacker wants. Somewhere in that pile there may be one real email that matters, like a bank alert or a password reset you did not ask for.

First, search your inbox for words like "payment", "login", "password", or your bank's name. Handle the real one first. Call your bank if you see a charge you did not make.

Then use the stop and unsubscribe links in the junk. Most real services put them there. It is slow work, but it cuts the flood at the source.

If it keeps coming, tell your email provider. Gmail and others can filter a flood once they see the pattern. And if someone calls you "from IT" right after the flood starts, be careful. That call can be part of the attack.

One last thing

I like uptime tools. I build one. But any tool that will email a stranger on request can be turned into a small weapon, and most builders never test for it. If you run one of these, spend an afternoon trying to bomb yourself. You will learn more from that hour than from any security checklist, and you will find the holes before someone else does.

More reading: how I built this monitor, and how to report abuse to us.

Building an uptime monitor in Rust: one binary, two databases, 130K checks/sec per core

Slim — Tue, 30 Jun 2026 14:22:09 +0000

I spent the last few months building Uptimepage, an open-source uptime monitor and status page written in Rust. This post is the build story: the decisions that shaped it, the parts I rewrote, and the numbers that came out the other side.

The whole thing ships as one self-contained binary of about 23 MB, plus Postgres and ClickHouse. You can docker compose up and self-host it, or use the hosted tier. Source is AGPL-3.0 on GitHub.

Why one binary

A status page is a small surface with a lot of moving parts behind it: a scheduler, probe workers, an HTTP client, a time-series writer, an incident detector, an alerting fan-out, a web UI, a JSON API. The usual answer is a handful of services and a message bus between them.

I went the other way. One process, one binary, the same image whether it runs the control plane or a remote probe. No queue to operate, no version skew between services, no "which container is wedged" at 3am. The cost is that you have to be careful about what shares a thread and what can stall what. Most of the engineering below is about keeping those boundaries clean inside a single process.

Stack: Rust 1.95 (edition 2024), Tokio, Axum 0.8, Askama for compile-time HTML templates, HTMX 2 for partial swaps so there is no SPA framework to ship. The API stays the single source of truth because every UI mutation hits the same /api/v1/* endpoint a script would.

Two databases, on purpose

Monitors are low-cardinality relational data that gets mutated by API calls: targets, regions, channels, incidents, plans. That is Postgres. Check results are append-only, high-cardinality, and almost always queried by time range. That is ClickHouse.

Trying to force one of those into the other is where uptime monitors usually fall over. Putting billions of check results in Postgres turns every dashboard query into a sequential scan. Putting your relational config in ClickHouse means fighting its update model forever. So: Postgres 18 for the world, ClickHouse 26.3 for the firehose. Both run their migrations at process startup, so there is no separate migrator to forget.

The HTTP client I did not want to write

The first version used a popular high-level HTTP client. It worked, but a monitor is a weird HTTP workload: you connect once per target per interval, you never reuse the connection, and you care about the timing of each phase more than the body.

So I dropped down to hyper and hyper-util with rustls and built a connector that times DNS resolution, TCP connect, and the TLS handshake as separate numbers, then runs the request over hyper::client::conn and aborts the connection task the moment the body is read. Each result carries dns_ms, connect_ms, tls_ms, and ttfb_ms as distinct columns, which is what makes "it got slow but it is the DNS, not your server" a thing the dashboard can actually say.

The rewrite paid for itself. On a single core the client sustains around 130K checks/sec at saturation, roughly 7.7 microseconds per check. That was a 44 to 56 percent throughput gain over the old path. A chunk of it was a url::parse call hiding in the redirect policy that cost 7.5 percent on its own and just vanished. Two cores get to about 153K. Scaling goes sub-linear past four cores because of shared HTTP/2 connection state and the pool mutex, which is a fine problem to have for this workload.

These numbers come from a laptop and a loadtest binary, so I treat them as regression detection, not capacity planning. The headline in-process run pushed 252K requests/sec sustained at p99 393ms. The design goal is around 50K concurrent in-flight checks per node with under 50ms p99 of per-check overhead on top of the network.

One heap, not a timer per target

The naive scheduler spawns a timer task per monitor. That falls apart at fleet size: thousands of tasks, thousands of wakeups, memory that grows with the number of targets.

Instead there is a single driver task that owns one BinaryHeap<Reverse<Due>>, a min-heap keyed by the next due Instant for the whole fleet. Memory stays flat in fleet size. Each target gets a deterministic jitter offset hashed from its UUID so a thousand monitors on a 60s interval do not all fire on the same tick. Generation and sequence counters mean a re-scheduled target supersedes its stale heap entry instead of double-firing. The registry refresh that pulls config from Postgres runs on its own task with exponential backoff, so a Postgres hiccup never stalls dispatch: the scheduler keeps running on what it last knew.

Failure isolation inside one process

Because everything is one process, one bad target cannot be allowed to take down the rest. Three patterns do most of that work. Per-host circuit breakers trip when a host keeps failing, so it fails fast with circuit_open instead of tying up a worker on a timeout, then probes half-open after a cooldown. A per-tenant host throttle bulkhead caps how many checks can be in flight against one host at once; over the cap, a check is recorded as throttled and degraded rather than piling on, and it never pages. And singleflight on RDAP collapses domain-expiry checks for the same domain across many tenants into one upstream probe, with sticky last-good state so a flaky registrar does not flip the monitor red.

The worker pool itself is a task-per-dispatch gated by a semaphore sized to a max-concurrency setting, with an SSRF guard filtering resolved IPs before any connect.

Modeling time series in ClickHouse

The check_results table is a MergeTree ordered by (org_id, target_id, region, timestamp). The org_id leads the sort key so each tenant gets a sparse-index slice, but it is deliberately kept out of the partition key (partition is by day) so we do not end up with millions of partitions.

The columns are where the storage savings live:

Timestamps are DateTime('UTC') with CODEC(DoubleDelta, ZSTD(1)). Check intervals are near-constant, so DoubleDelta crushes the gaps to almost nothing.
Numeric phase columns (duration_ms, dns_ms, connect_ms, tls_ms, ttfb_ms, response code and size) use CODEC(T64, ZSTD(1)).
region and agent_id are LowCardinality(String), status is an Enum8.

Retention is per row. There is a ttl_days column with TTL timestamp + toIntervalDay(ttl_days), stamped from the org's plan at write time. A plan that buys longer history needs zero schema change: new rows just carry a bigger number.

On top of the raw table sit two AggregatingMergeTree materialized views: a per-minute rollup kept 30 days and an hourly rollup kept 13 months, both holding quantilesState for p50/p95/p99 and per-status counts. Reads route by range: anything inside 30 days hits the minute rollup, older ranges hit the hour rollup, and raw reads are capped at a 90-day span. A dashboard asking for "last 24h p99 latency" never touches a raw row.

Regional probes without a second brain

You can run probes in multiple regions, but I did not want each region to be its own little system. So an agent is the same binary in agent mode, running as a stateless probe with no database, no web, no alerting. Adding a region adds execution capacity, never a second control plane.

An agent pulls its region's config from the control plane with ETag/304 handling, serves its last-known config if the control plane blips, and pauses if its token is revoked. It ships results back in batches that reuse one UUID across retries so a lost ack cannot double-count. Region and agent identity are derived server-side from the bearer token, never sent in the payload, so a probe cannot claim to be somewhere it is not. A separate long-poll loop handles interactive "check now" so a button press in the UI runs on a real remote probe within milliseconds.

Region is the partition dimension end to end, so every read can slice by region: per-region latency series, per-region incident scope, "down in Singapore, up in Helsinki."

Incidents as a follower, not a gatekeeper

The incident detector is a background task that follows the check_results stream and writes into the Postgres incidents table. The rule I held to: it never touches the hot write path, never gates check execution, and never produces alerts directly.

The detection itself is boring on purpose: two or more consecutive unhealthy results with no open incident opens one, two or more consecutive healthy results closes it. A small flap threshold absorbs single-result blips. The cross-tenant walk is keyset-paginated so memory stays bounded as tenants grow, and a unique index on open incidents resolves the race when two ticks try to open the same one: only the winner pages.

Opening or resolving fires a non-blocking signal to the escalation engine, which does repeat-until-acknowledged paging on a per-monitor cadence across about fourteen transports (Slack, generic signed webhooks, Telegram, PagerDuty, ntfy, Pushover, Discord, email, and more), with sharded per-incident locks so a reconcile and an inbound signal can never double-page the same episode. Channel secrets are sealed at rest with AES-GCM and never echoed back.

Automation as a first-class surface

Because the API is the single source of truth, the rest came almost for free:

A self-describing OpenAPI 3.1 spec with Swagger UI.
An official Terraform provider so you can manage monitors and notification channels as code.
An MCP server so an LLM client can query your monitors and incidents over OAuth 2.1, scope-gated and audited, with per-action confirmation on the few write tools.

Where it is

Uptimepage is live, free to start with no card, and AGPL-3.0 open source. The core is not paywalled: checks, status pages, subscribers, the API, and every alert channel are in the free tier. It monitors itself and serves its own status badges from the running binary, which is the most honest dogfood I could think of.

Site and hosted tier: https://uptimepage.dev
Source: https://github.com/uptimepage/uptimepage
More build notes: https://uptimepage.dev/blog

Happy to answer anything about the Rust internals in the comments.