DEV Community: Slim

Uncle Bob's uml-viewer on Rust: 35 dependency cycles down to 3

Slim — Tue, 22 Sep 2026 06:58:19 +0000

TL;DR. Uncle Bob released uml-viewer, a clickable architecture diagram that draws Dependency Rule violations in red and is meant to be driven together with a coding agent. It only parses Clojure, so I wrote a 200-line script that turns my Rust crate into its input file. The first picture showed 167 red arrows and 35 dependency cycles. Six commits and 315 files later: 109 red arrows, 3 cycles, and app.rs down from 1,087 lines to 642. No behavior changed. The tool matters less than the loop it forces on you: look, point at one arrow, let the agent move code, look again. The original post has the same text with a collapsible FAQ and the source list at the end.

Uncle Bob's argument for the tool is short: agents still need supervision, and reading every line they write is the bottleneck. So supervise the structure instead of the text: draw the system, find the shape that is wrong, tell the agent to fix that shape, and check the new drawing.

I wanted to know if that works on a real codebase, so I pointed it at Uptimepage, about 146,000 lines of Rust in one crate. It does, with one condition: the picture is only as honest as the input file you feed it.

Five of the 167 red arrows and two of the 35 cycles, and where the code went. Every arrow that pointed up now points down.

What uml-viewer is

uml-viewer is an open-source desktop tool by Uncle Bob (Robert Martin) that draws a codebase as a UML-like diagram you can click. It is written in Clojure and needs the Clojure CLI and Java 21 or newer.

Namespaces are components. The modules inside a namespace are the component's elements. Nesting can go as deep as your source tree does. You double-click a component to open the next level, double-click a module to see its class card, and click a function to open the source file at that line.

Color comes from CRAP and mutation scores, so a red box is code with high complexity and weak tests, or code with no metrics at all, which the README counts as the worst grade. Red arrows come from the Dependency Rule: you tell the tool which namespaces sit at which architectural level, and every dependency that points from an inner level to an outer level is drawn red.

The tool is built to run next to an agent. By default it opens a tmux window with Grok in the examined project and the two talk through a small mailbox directory. You can also drive it by hand from any agent session: regenerate the input file, press R in the viewer, and the diagram reloads.

Getting a Rust crate into it

The only parser it ships reads Clojure. That sounded like the end of the experiment, until I read what the viewer actually consumes. It never sees source code. It reads one EDN file: a list of classes with a namespace and a level, a list of edges with a from, a to and a kind, and a list of levels. That is a format any script can emit.

So I asked the agent to write a generator for Rust. It came out at about 200 lines of Python and does four things:

Every .rs file is one class. mod.rs stands for its directory and lib.rs is skipped.
Every use crate::…, super::… and self::… path, plus every inline crate::a::b path in a function body, becomes a dependency edge to the nearest enclosing module that exists as a file.
Comments are stripped first, so a path mentioned in a doc comment does not count as a dependency.
A hand-written LEVELS list gives each top-level module a rank.

The last item is the only opinion in the script:

# inner (high level) first, matching the Dependency Rule ranks
LEVELS = [
    ["domain", "error", "text", "metric_names"],
    ["storage", "security", "net", "http_client", "config",
     "quotas", "observability", "pagination"],
    ["pipeline", "escalation", "notifier", "email", "telegram",
     "whatsapp", "billing", "auth", "analytics",
     "jobs", "http_outbound", "targets"],
    ["api", "web", "mcp", "marketing", "agent", "worker", "scheduler",
     "request", "public_status", "templates", "oauth",
     "ad_hoc_dispatch", "channels"],
    ["app", "router", "bootstrap", "main", "bin"],
]

Rank 0 is vocabulary that everyone may use: domain types, error codes, text helpers. Rank 1 is infrastructure: storage, security primitives, config. Rank 2 is the services that do the work: probing, escalation, notification, billing. Rank 3 is every way into the system: HTTP handlers, HTML views, the MCP server, the marketing site. Rank 4 is assembly: the app state, the router, main.

The rule is then mechanical. An edge is a violation when the module it comes from has a smaller rank than the module it points to. Same rank is allowed. Foreign crates like axum and sqlx are ovals outside the diagram and are never compared.

The picture is only as honest as the levels list

The generator does not decide the architecture. I do, in that list. If you put api in the same group as domain, the diagram turns green and you have learned nothing. Write the list you want to be true, then let the red arrows show how far the code is from it.

What the first picture showed

The first diagram had 167 red arrows, 35 pairs of modules that imported each other, 17 modules importing api, and an app.rs of 1,087 lines.

Behind the numbers were shapes I half knew about and had never seen drawn:

storage reached up into api for error codes and dashboard read models, and into worker for heartbeat state.
api and web imported each other. The JSON side took session and token extractors, cookies and client IP from the HTML side, and the HTML side took the heartbeat read model from the JSON handlers.
quotas read accounts and organizations from storage, while storage embedded quota SQL fragments and read plan types from quotas. A cycle in both directions.
marketing and oauth imported web for three template filters.
config imported auth and billing for two enums.

None of this was visible in the review of any single pull request. Every one of those imports was reasonable on the day it was written. A diff shows one import at a time, so the sum of them never appeared in any review.

The loop

Every round had the same six steps:

Regenerate the EDN file and reload the viewer.
Pick one red arrow. Hover it to see which module pairs it bundles.
Tell the agent one thing: what must not import what, and where the shared piece should live.
The agent moves the code and runs the compiler and the tests.
Regenerate. Check that the arrow is gone and count the new ones.
Run the full suite, review the diff, commit.

The instruction in step 3 is the part that makes this work. It looks like this:

storage must not import api. The error codes and the dashboard read models
it takes from there are crate-wide vocabulary. Move the codes to error::codes
and the read models to domain::metrics, and point every reader at the new
path. No behavior changes.

That is one rule, one violation, and one place to put the result. The agent does not have to guess what "cleaner" means. It has a named arrow to remove, and the next diagram says whether it did.

I ran this with Claude Code, but nothing in the loop depends on it. Codex, Cursor, Grok, or any agent that can edit files and run a test suite gets the same instruction and the same picture afterwards.

Six rounds took the crate from 167 red arrows to 109:

Round	Red arrows I pointed at	Where the code went
1	`storage` into `api` and `worker`, `config` into `auth`	error codes to `error::codes`, 12 read models to `domain::metrics`, provider enums to `domain::credential`
2	`storage` into `auth` and `api`	token hashing, HMAC and SHA-256 helpers to `security`, redaction scrubbers to `security::redaction`
3	`api` and `web` into each other	extractors, cookies, client IP and host resolution to a new `request` module that imports neither
4	`storage` and `quotas` into each other	SQL fragments to `storage::count_sql`, plan types to `domain::quota`
5	`marketing`, `oauth` and `api` into `web`	template filters and formatters to `templates`, `app.rs` split into `config::boot`, `observability::readiness`, `targets::status` and `net`
6	`public_status` and `web` into `api`, `security` into `http_client`	page envelopes to `pagination`, `Cipher::from_config` into `security`

And the totals:

Six commits, 315 files, same test suite before and after. Violations down 35 percent, cycles down 91 percent.

	Before	After
Dependency Rule violations	167	109
Module pairs importing each other	35	3
Modules importing `api`	17	4
Lines in `app.rs`	1,087	642
Files touched		315
Behavior changes		0

I regenerated the graph at every one of the six commits to see what each round removed:

Cycles and api importers per round. The first two rounds did most of the work on the red count; rounds three to six were about the cycles.

The shape surprised me. The red arrow count stopped moving after round two. Rounds three to six barely touched it, but they took the cycles from 23 down to 3, because a cycle between two modules at the same level is not a Dependency Rule violation at all. The rule catches arrows that point up and says nothing about two modules on the same level that import each other, so you need both counts.

Not every move survived. In a later round the agent moved the health-check paths into observability, and a coupling test on the marketing module said no, because that module is only allowed to reach a short list of leaf modules. Another move put the strict JSON parser under request, then had to come back because that parser reads the OpenAPI document. Both reverts took minutes, because the diagram and the test said so before the commit did.

Why I stopped at 91

After the six commits, 109 red arrows were left. Most pointed into app, but ten did not: a sampler and a silence job that lived under observability but reached into the scheduler and the notifier, two background jobs reading public_status, the HTTP metrics layer, and the rate-limit middleware in quotas reading the app state. That middleware was also the third of the three cycles. One more round moved each of those into the module that owns it and left 91 red arrows and 2 cycles.

Every one of the 91 is the same shape. Handlers read the app state, and the app state is built from the modules those handlers live in. app imports api and web to mount them; api and web import app to get the state.

Fixing it is possible. Split the state into per-handler sub-states and hand each handler only its slice. I counted what that would touch: about 90 files of plumbing that make the code harder to read, not easier. I stopped because I could name every remaining arrow, and at that point the diagram had told me everything it was going to.

Stop when every arrow has a name

Drive the count down until every arrow that is left has a name and a reason, then stop. The 91 arrows left in my crate are all "this handler reads the app state", and a diagram that shows them is more honest than one that hides them behind 90 files of indirection.

Keeping it fixed

Two things stop the graph from drifting back.

The marketing module has a test that is an allow-list: the exact set of leaf modules it may import. A new reach into the app fails the build. It used to be a deny-list, which only catches the mistakes you already thought of.

The other is a habit, not a test. Regressions arrive with feature commits, not with refactors. After every push I regenerate the graph and read the list of red arrows that do not point into app. If the list is empty, the feature stayed in its layer. If it is not, the offending edge is one instruction away from gone, before the next feature builds on it.

Why this beats "refactor the codebase"

Two months ago I wrote about mapping this codebase for humans and AI agents. The lesson then was that a model is good at shape and bad at numbers, so I had to check every count it produced by hand.

This is the same lesson, applied to refactoring. Here the agent never counts anything; the generator does. The agent gets one rule with one violation and a picture that says pass or fail after every change. It is the same reason a failing test is a better instruction than a paragraph of requirements: the acceptance criterion exists before the work starts, and it is not the agent that judges it.

What I did not get is the color. CRAP and mutation scores come from Uncle Bob's Clojure tooling, and there is no Rust equivalent wired in yet. The README is clear that a box with no metrics is painted as the worst grade, so the fill on my boxes means nothing until coverage and mutation numbers exist for Rust. The arrows alone were worth the setup.

Do it yourself

Install the Clojure CLI and Java 21 or newer, then clone uml-viewer.
Write a generator for your language. One class per module, one dependency edge per import, output as EDN with :hierarchical true. The generated examples/uml-viewer.edn in the repo is a complete example of the format. In Rust, imports are use paths; in TypeScript they are import statements; in Python, import and from. A regex over each file is enough to start.
Write the :levels list by hand, inner layer first. Describe the architecture you want; the red arrows will show where the code differs.
Run the viewer against your file. On a fresh start it waits for its companion agent; the --restart flag restores the last view without spawning one, which is what you want when you drive it from your own agent session.
Pick one red arrow. Write one instruction that names the two modules and the new home. Regenerate. Repeat.

Key takeaways

uml-viewer reads a plain EDN file, not source code. A 200-line script gets any language in.

The levels list is the only opinion in the input, so write the architecture you want and let the red show the distance.

Give the agent one rule, one violation and one destination per instruction. The next diagram is the acceptance test.

Rust will not tell you two modules import each other. Build the graph and count the mutual pairs yourself.

Stop when every remaining red arrow has a name. Mine are all "handler reads app state", and that is fine.

Turn the result into an allow-list test, and regenerate the graph after every feature push, because that is when regressions arrive.

The six commits are public in the Uptimepage repository if you want to read what moved and why.

Sources

Robert C. Martin, uml-viewer on GitHub. The README documents the EDN format, the :levels rule and the companion agent mailbox.
Robert C. Martin, The Clean Architecture, The Clean Code Blog, August 2012. The Dependency Rule.
Uptimepage, source repository, AGPL. Commits bda59342 through cee12c44 are the six rounds described here.

I build Uptimepage because I wanted uptime checks that say why something failed instead of "transport error": HTTP with per-phase timing, TLS and domain expiry, ping and TCP, heartbeats for background jobs, and browser flows, from several regions. It is AGPL-3.0 open source, which is why the six commits above are public and you can check every number in this post against them.

Now the question for the comments: what draws the module graph for your language, and does it show the cycles or only the layer violations? My rounds three to six only moved the cycle count, and I would have missed that with a tool that colors arrows but never counts pairs.

ICMP vs TCP vs UDP: the difference, explained for developers

Slim — Sun, 13 Sep 2026 09:49:11 +0000

Originally published on the Uptimepage blog. I build an uptime monitor in Rust, and its ping, TCP and DNS checks are these three protocols with a timer on them, so I had to get the differences straight.

TL;DR. ICMP, TCP and UDP all travel inside IP packets, but they ask different questions. ICMP echo (ping) asks "does this address answer at all?" and needs no port. TCP asks "is a program listening on this port?" and gets a clear yes, a clear no, or silence. UDP asks nothing by itself: you only learn something if the program on the other side chooses to reply. A closed port answers differently on each one, and that difference is most of what a network check can and cannot tell you. The original post has the same text with a collapsible FAQ and the RFC list at the end.

One address, three ways to knock

Every packet on the internet is an IP packet. IP knows how to move a packet from one address to another and nothing else. One byte in the IP header says what is inside, and IANA keeps the list: ICMP is protocol 1, TCP is protocol 6, UDP is protocol 17. The three are siblings: same addresses, same routers, and a different promise about what happens once the packet arrives.

Three ways to knock on one address. The last row of each lane is what silence means there.

	ICMP	TCP	UDP
Specification	RFC 792 (1981), RFC 4443 for IPv6	RFC 9293 (2022)	RFC 768 (1980)
IP protocol number	1 (58 for ICMPv6)	6	17
Ports	none	source and destination	source and destination
Handshake	none	three packets before any data	none
Delivery guarantee	none	ordered, complete, retransmitted	none
Header	8 bytes for an echo	20 bytes minimum	8 bytes
Typical users	ping, traceroute, error reports	HTTP/1.1 and HTTP/2, TLS, SSH, SQL, SMTP	DNS, NTP, QUIC and HTTP/3, video, games

ICMP: the network talking about itself

ICMP is not a transport. RFC 792 says it plainly: ICMP "is actually an integral part of IP, and must be implemented by every IP module." It exists so that routers and hosts can report on delivery: this destination is unreachable, that packet ran out of hops. ICMP messages have a type and a code but no ports, because nothing sits on top of ICMP waiting for them. The IP stack consumes them itself.

The one ICMP message everybody has typed is the echo. ping sends an echo request (type 8, or 128 in ICMPv6) and the far host's kernel sends back an echo reply (type 0, or 129). No program had to be running. RFC 1122, the host requirements standard, makes it a rule: "Every host MUST implement an ICMP Echo server function that receives Echo Requests and sends corresponding Echo Replies."

$ ping -c 1 uptimepage.dev
PING uptimepage.dev (204.168.246.94): 56 data bytes
64 bytes from 204.168.246.94: icmp_seq=0 ttl=50 time=96.069 ms

A reply proves two things: the address belongs to a machine that is switched on, and packets can travel there and back. It also gives you the round-trip time for free.

That is all a ping proves. It says nothing about whether a web server, a database or anything else is running, and a host can answer every ping while every service on it is dead.

Two things about ICMP surprise people. The first is that no reply does not mean down. Firewalls drop ICMP all the time. On AWS a fresh security group blocks it, and the documentation says so: "To ping your instance, you must add one of the following inbound ICMP rules." Before you rely on ping for a host, confirm the host answers a ping at all.

The second is that sending ICMP needs privilege. A raw socket needs CAP_NET_RAW. Linux has an unprivileged alternative, the ICMP datagram socket, but the kernel documentation says the default ping_group_range is "1 0", "meaning, that nobody (not even root) may create ping sockets." Docker opens that range in every container that gets its own network namespace, which is why ping works there without anyone thinking about it; a container on --network host inherits the host's setting instead. I learned the limits of that the slow way: the same probe binary that pinged fine in Docker could not open an ICMP socket at all on a Firecracker microVM, until I gave it CAP_NET_RAW.

TCP: a promise before the first byte

TCP is a connection. Before either side sends a byte of data, the two ends run the three-way handshake: SYN, SYN-ACK, ACK. RFC 9293, which replaced the 1981 specification in 2022, calls it "the procedure used to establish a connection." It costs one round trip, and that round trip pays for every promise TCP makes afterwards: bytes arrive in order, lost segments are sent again, the receiver is not flooded (flow control), and the network is not flooded (congestion control). To the program it looks like a stream. You write bytes on one side and read the same bytes, in the same order, on the other.

This is what HTTP/1.1, HTTP/2, TLS, SSH, PostgreSQL, MySQL, SMTP and most of what you deploy run on.

The handshake is also a good probe, because a SYN gets one of three answers:

SYN-ACK. A program is listening. The connect succeeds.
RST. Nothing is listening. RFC 9293 says that if the connection does not exist, "a reset is sent in response to any incoming segment except another reset." Your connect fails at once with "connection refused".
Nothing. A firewall dropped the SYN, or the host is gone. The kernel keeps retrying. On Linux the default tcp_syn_retries is 6, and the kernel documentation puts the final timeout for an active connection attempt at 131 seconds.

Here are all three from my laptop. The last one has a three second limit; without it macOS waits 75 seconds.

$ nc -zv uptimepage.dev 443
Connection to uptimepage.dev port 443 [tcp/https] succeeded!

$ nc -zv 127.0.0.1 9
nc: connectx to 127.0.0.1 port 9 (tcp) failed: Connection refused

$ nc -zv -G 3 uptimepage.dev 8443
nc: connectx to uptimepage.dev port 8443 (tcp) failed: Operation timed out

Nmap's documentation gives these states the names everyone uses: open, closed and filtered.

What a successful handshake proves: a process accepted a connection on that port. Not that it is healthy. A database that accepts connections and then rejects every query still passes. That is why a TCP check belongs on things that have no HTTP surface (SSH, SMTP, a database port) and an HTTP check belongs on things that do.

UDP: a message, nothing more

UDP is the smallest transport there is. RFC 768 fits on three pages. It adds a source port, a destination port, a length and a checksum to an IP packet, 8 bytes in total, and stops. The RFC says: "The protocol is transaction oriented, and delivery and duplicate protection are not guaranteed." There is no handshake, no acknowledgement, no ordering and no retransmission, and no congestion control either. RFC 8085 opens by saying UDP "has no inherent congestion control mechanisms" and then spends more than fifty pages telling application designers how to add their own.

Why anyone uses it: latency and control. A DNS lookup is one packet out and one packet back, no handshake. RFC 1035 capped a DNS answer over UDP at 512 bytes. EDNS(0) (RFC 6891) later let a client advertise a bigger buffer, so most answers fit in one datagram today; one that still does not fit is cut short with the TC bit set, and the client asks again over TCP (RFC 7766). NTP, most game traffic and real-time video use UDP because a late packet is worth less than the next one, so waiting for a retransmission is the wrong trade. And QUIC, the transport under HTTP/3, is built entirely on UDP. RFC 9000 states that "QUIC packets are carried in UDP datagrams" and gives the reason as "to better facilitate deployment in existing systems and networks". In practice that means QUIC does its own handshake, loss recovery and congestion control in the application, and can change them without waiting for every operating system to update its TCP stack.

$ dig @1.1.1.1 uptimepage.dev A +noall +answer +stats
; <<>> DiG 9.10.6 <<>> @1.1.1.1 uptimepage.dev A +noall +answer +stats
; (1 server found)
;; global options: +cmd
uptimepage.dev.     3600    IN  A   204.168.246.94
;; Query time: 125 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Wed Sep 02 11:40:44 EEST 2026
;; MSG SIZE  rcvd: 59

One datagram out, one datagram back carrying a 59-byte answer (67 bytes with the UDP header), and no handshake before it. Over TCP the same lookup costs two round trips, one for the handshake and one for the query, which is why DNS defaults to UDP.

Now the probing problem. Send a UDP datagram to a port and what comes back?

If nothing listens there, the host's IP stack should answer with an ICMP error. RFC 1122 again: "If a datagram arrives addressed to a UDP port for which there is no pending LISTEN call, UDP SHOULD send an ICMP Port Unreachable message." So "closed" is detectable, and it is ICMP that detects it.

If something listens, you get whatever that program chooses to send. A DNS server sends an answer. A syslog receiver sends nothing, ever. So silence means open, or filtered, or dead, and Nmap has a state for this, open|filtered. Its documentation explains: "This occurs for scan types in which open ports give no response."

The consequence is that a generic "UDP port check" does not exist. To check a UDP service you have to speak its protocol: send a real DNS query and read the answer, send a real NTP request and check the timestamp. This is why monitoring tools have a DNS check and a TCP check but rarely a UDP check.

The same closed door, three answers

Situation	ICMP echo	TCP SYN	UDP datagram
Service up	echo reply	SYN-ACK	the program's reply, or nothing
Port closed, host up	echo reply (ICMP has no ports)	RST, "connection refused"	ICMP port unreachable
Firewall drops the packet	silence	silence, retried for about two minutes	silence
Host gone	silence, or an unreachable error from the last router	silence, or that same ICMP error surfacing as "no route to host"	silence, or that same ICMP error

Read the table by columns. TCP is the only one where "closed" and "filtered" look different on their own, UDP borrows ICMP to say "closed", and ICMP cannot see ports at all. Silence is the ambiguous answer on all three, which is why no check should conclude "down" from one silent probe sent from one place. Two or three consecutive misses, seen from more than one network, is the honest bar. I wrote about that in how to stop false uptime alerts.

What this means for a monitor

I build Uptimepage, and its check types map onto these questions, so this is how the theory turns into a config.

Ping sends one ICMP echo with a 32-byte payload, the size Windows ping uses (Unix ping sends 56, as the transcript above shows), so middleboxes see ordinary diagnostic traffic. It tries the first IPv4 and the first IPv6 address of the host and splits the timeout between them. Silence for the whole budget is down, because an echo has no way to refuse. Use it for routers, gateways and hosts that expose no service.
TCP runs the handshake and closes the connection. Accepted is up, refused is down, and a timeout is recorded as an error with the time it took. An error counts as a failed check for alerting exactly like down, so a firewalled port still opens an incident once the confirmation threshold is met. Use it for databases, brokers, SSH and mail, where "a process accepted the connection" is the most you can assert from outside.
DNS sends a real query and checks the answer's content, so it is the UDP check for the one UDP service almost everyone depends on. It can point at a specific resolver, which matters when you want that server's view and not your cache's.
HTTP is a TCP handshake, then TLS, then a request, and it records the time spent in each phase: DNS, connect, TLS and time to first byte. The connect phase is the handshake from this article, measured inside a bigger check.

Pick the check that asks the question you care about, and remember what each one cannot see. There is a fuller version of that decision in the monitor types documentation, and a story-shaped version in the mystery of the "down" website.

Common questions

Is ICMP a transport protocol like TCP and UDP?

No. ICMP is a control protocol that belongs to IP itself, and RFC 792 says every IP module must implement it. It has no ports and carries no application data. It reports on delivery: a destination is unreachable, a packet ran out of hops, and the echo pair that ping uses.

Why does ping work while my website is down?

Because ping only proves that the host's IP stack answers and that packets can travel there and back. The kernel sends the echo reply, so no web server, database or other program has to be running. A host can answer every ping while every service on it is dead.

Why does a closed TCP port fail at once but a firewalled one hangs?

A closed port sends back a TCP reset, so the connect fails immediately with connection refused. A firewall that drops the SYN sends nothing, and the kernel keeps retrying. On Linux the default is six retries, and the kernel documentation puts the final timeout at 131 seconds.

Can I check a UDP port the way I check a TCP port?

No, because an open UDP port sends nothing back unless the program behind it chooses to answer. A closed port is detectable, since the host should reply with an ICMP port unreachable message, but open and filtered both look like silence. To check a UDP service you have to speak its protocol, for example send a real DNS query.

If UDP is unreliable, why does HTTP/3 use it?

UDP itself gives no delivery guarantee, and QUIC adds its own on top. QUIC runs inside UDP datagrams and does its own handshake, retransmission and congestion control in the application, which lets it evolve without waiting for every operating system to update its TCP stack.

Sources

IETF, RFC 792: Internet Control Message Protocol, September 1981.
IETF, RFC 4443: ICMPv6 for the Internet Protocol Version 6, March 2006.
IETF, RFC 768: User Datagram Protocol, August 1980.
IETF, RFC 9293: Transmission Control Protocol, August 2022.
IETF, RFC 1122: Requirements for Internet Hosts, Communication Layers, October 1989. Sections 3.2.2.6 and 4.1.3.1.
IETF, RFC 8085: UDP Usage Guidelines, March 2017.
IETF, RFC 1035: Domain Names, Implementation and Specification, November 1987. Section 4.2.1.
IETF, RFC 6891: Extension Mechanisms for DNS (EDNS(0)), April 2013.
IETF, RFC 7766: DNS Transport over TCP, Implementation Requirements, March 2016.
IETF, RFC 9000: QUIC, a UDP-Based Multiplexed and Secure Transport, May 2021.
IANA, Assigned Internet Protocol Numbers.
Linux kernel, IP sysctl documentation, tcp_syn_retries and ping_group_range.
Nmap, Port scanning basics, the six port states.
Amazon Web Services, Security group rules for different use cases, rules for ping/ICMP.

I build Uptimepage because I wanted these checks for my own services: ping, TCP, DNS and HTTP with per-phase timing, plus TLS and domain expiry, heartbeats for background jobs, and browser flows, from several regions. It is AGPL-3.0 open source, so you can run it on your own server if you prefer.

Now the question for the comments: do you still run ping checks on anything, and what for? I keep them for routers, gateways and hosts that expose no service, and almost nothing else. If a ping check has earned its keep for you somewhere I have not listed, I would like to hear where.

Your domain can expire while your uptime monitor stays green

Slim — Tue, 04 Aug 2026 14:15:27 +0000

TL;DR. One HTTP check on your homepage is not enough to know your site works. It cannot tell you when your domain expires, because an expired domain still returns 200 OK from a parking page. Mail dies at the same time, just as quietly, and you keep losing money while every monitor stays green. The fix is not one clever check. It is a few plain ones, starting with the two that fail on a known date: domain and TLS expiry.

Two checks green, the server behind them is down. A status check cannot tell the difference.

I keep seeing new products treat reliability as something that takes care of itself. You ship the site, you watch it load once, and you assume it will keep loading. Hosted once, up forever. That is the most expensive assumption a young team can make, because nobody is watching on the day it stops being true.

So ask one honest question. Is one HTTP check on your homepage enough to know your whole site works?

For most teams it is not. Here is a real case that shows why.

The site was gone, but it answered 200 OK

I was looking at a startup's site last week and it would not load right. Not an error, not a timeout, just the wrong page. The domain had expired two days earlier.

When a domain lapses, the registrar does not switch the site off. It points the domain at its own parking page, the "this domain may be for sale" one. That page is a normal web page. It returns 200 OK and loads fast. It stays that way for a while too, because an expired domain is not deleted the next morning.

So the monitor sees a healthy site. Green light, no alert, everyone asleep. What could be wrong?

Your domain expired. You knew the date once, on the day you bought it a year or two ago. You told yourself you would renew closer to the time, and then you forgot. The card on file lapsed, or the reminder went to an inbox you no longer read.

Now your customers cannot get in. They panic. They email to ask whether you are still in business. And they get no answer, because the mail died with the domain too.

Why the mail dies with it

The registration carries your MX records, so when it lapses, mail to every address on the domain starts bouncing. The website looks up and the inbox goes silent, and the two failures do not obviously connect. That is why the angry email never reaches you.

The front page still answers. Behind it, nothing is running.

And the meter keeps running. While the page says 200, your ads keep sending paid clicks to a parking lander, signups never arrive, and recovery is not fully in your hands. Two days down is not a rounding error either, because downtime adds up faster than the percentages suggest. Once a domain enters redemption after expiry, getting it back can cost far more than a renewal, and the registry sets the clock, not you.

The clock after expiry

For a gTLD like a .com, the sequence is roughly this:

Grace period. The registrar parks the domain. The site is gone but the name still answers, usually for around 30 days, and a normal renewal still works.
Redemption, 30 days. Now it really goes dark. You can still get it back, but through a restore fee that is many times the renewal price.
Pending delete, about 5 days. Nothing you can do. At the end it drops to whoever is waiting.

ICANN's renewal and expiration FAQ and its grace period notes have the exact stages. The practical takeaway is that the first month, the one where an uptime monitor is happily reporting 200, is the cheap month. Everything after that costs money or the domain.

My own uptime check would miss it too

I build an uptime monitor, so here is the uncomfortable part. My own HTTP check would have called that dead site up too. A 200 is a 200. A status-code check cannot tell a real page from a parking page, and it should not pretend that it can.

That is the trap. One HTTP check on the homepage, the most common setup there is, is blind by design. The site answers, the code is green, and the part that pays your bills is broken behind it.

You can bolt on body matching, and it helps a bit. Assert that the response contains a string only your app renders, and a parking page fails it. But a keyword check is still guessing at the cause after the fact. The registration date is a fact you can read months in advance, so read it.

One check is not enough, set up several

So how do you avoid reading about your own outage in a customer's email? Not with one cleverer check. With a few plain ones, each watching a different way to fail. Start with the two that hand you a date instead of a surprise.

Domain expiry. Reads the registration record and warns you weeks before the day it lapses.
TLS certificate expiry. The same shape, a full outage with the date printed on it in advance.

Then cover the failures a homepage check never sees: a TCP check on your database port, a DNS check that reads the answer from outside your own network, and a heartbeat for the backup job that dies without a sound. The full list is in do I need an uptime monitor.

Six checks you trust beat one that only ever watches the front door.

How to set up domain expiry with an alert

Two minutes now buys you weeks of warning later. The shape is the same in any tool that supports it:

Add a monitor and choose the Domain expiry type.
Enter the registered domain, for example yourbusiness.com, not a subdomain.
Set the thresholds in days. A common choice is a warning at 30 days and a critical alert at 7.
Attach a notification channel you actually read. Telegram, Slack, PagerDuty, SMS, a webhook, or email all work. Pick the one that reaches you at night.

From then on it checks once a day, warns you when the date crosses your line, and reminds you as it gets close. No page-watching, no false green.

If you would rather not add a service for it, a cron job that runs whois and greps the expiry date will get you most of the way. The point is the date, not the tool. What you must not do is leave the only reminder in a registrar email going to a founder's old inbox.

Key takeaways

One HTTP check on the homepage is not enough. It stays green while the domain, the certificate, DNS, or a background job is broken.

An expired domain serves a parking page that returns 200 OK, so status-code monitors report it as healthy.

Email dies at the same time and even more quietly, because the MX records lapse with the registration.

You keep losing money the whole time: paid clicks to a dead page, lost signups, and a slow, costly recovery.

Add the checks that fail on a known date first, domain and TLS expiry, then the rest.

Common questions

Why does an expired domain still return 200 OK? The registrar parks it on a lander page instead of taking it offline. The old server is gone, but the parking page is a real web page that answers normally, so a check that only reads the status code sees success.

How long before an expired domain is gone for good? You usually get a month or two. Grace period, then a 30-day redemption where the site is dark but recoverable for a fee, then a short pending-delete stage before it is released to anyone.

What else breaks when a domain expires? Your email, at the same moment and more quietly. The MX records go with the registration, so mail starts bouncing while you are still looking at a page that loads.

Is a keyword check enough instead? It catches the parking page, which is better than nothing, but only once you are already down. The expiry date is knowable months ahead. Use both if you like, but do not skip the date.

I build Uptimepage because I wanted checks like this for my own services: domain and TLS expiry, HTTP, TCP, ping and DNS, heartbeats for background jobs, and browser flows for logins, from several regions. It is AGPL-3.0 open source, so you can run it on your own server if you prefer.

Now the question I actually want answered in the comments: what is the dumbest green check you have been burned by? I have seen a health endpoint that returned 200 with a hardcoded {"status":"ok"} while the database behind it was unreachable for six hours. I suspect yours is better.

How I stop one bad probe from waking you at 3 a.m.

Slim — Tue, 28 Jul 2026 14:55:48 +0000

TL;DR. The most common false alert in uptime monitoring is one probe location having a bad network day. I treated that as a design requirement from day one, so the checking pipeline has two gates: a region only counts as down after it fails the same check twice in a row, and an incident only opens when enough regions agree. A region that goes silent leaves the vote instead of counting as down. One bad location cannot page you. The original post carries the three figures below as live widgets you can drive yourself.

The alert that was nothing

The phone buzzes at 3 a.m. "Your API is down." You get up, open the laptop, and everything is green. The site was fine the whole time. One monitoring server, somewhere far away, had a bad network moment and sent an alert about nothing.

This is the false alert everyone in monitoring knows. It costs you sleep, and then it costs you something worse: the next alert feels less serious. The day a real outage comes, you look at your phone and think "probably nothing again."

I knew this failure mode before I wrote the first line of the scheduler, so it became a requirement, on the same level as "checks must run on time." The rule I started from: a single bad location must never be able to page a customer. The whole checking pipeline, from how probes report to how incidents open, is shaped by that rule.

The shape it took is two gates. A failure has to pass both before anyone gets paged.

Gate one: fail twice, from the same place

One failed check proves very little. Networks lose packets, routers restart, and sometimes a DNS answer arrives a second too late. All of that can make one check fail while your site is healthy.

So a region only counts as down after it fails the same check twice in a row. Not two failures somewhere in the system: two failures from that one region, back to back. A single blip resets to zero on the next good check.

One blip resets the counter. Two failures in a row from the same region earn that region a single vote.

The count is a setting on each monitor. Two is the default. For a very sensitive check you can set it to one, and for a noisy target you can raise it. Checks run on the monitor's own interval, so with a one-minute check the second failure arrives about a minute after the first. That minute buys you a lot of silence for a very small delay.

Gate two: the vote

Passing gate one gives a region exactly one vote. Nothing more.

Say you check from five regions and one of them has a bad ISP day. It fails twice in a row and votes "down." The other four keep passing. One vote against four is not enough, so nothing happens and nobody gets paged.

One region against four cannot page you. Three of five can.

The default rule is a majority: more than half of the reporting regions have to agree. You can change it per monitor. "Any" opens an incident on the first confirmed region, which is useful when you want the earliest possible signal and can accept some noise. "All" waits until every region agrees, for a service that only matters when it is unreachable everywhere. Or you pick a fixed number, like two regions out of whatever you assign.

A monitor checked from a single region behaves the same under every rule. The vote only starts to protect you when you add a second location, and it gets better with a third.

Silence is not failure

Here is the part that took the most care to get right.

The probe regions push their results to the control plane, the brain of the system. The brain never calls out to ask "are you alive?" during a vote. It just counts the results that arrived in the last few check cycles.

That gives silence a clear meaning. A region that lost its own connection cannot send anything, so it simply is not in the vote. It is not a down vote and not an up vote; the region is out until it reports again. The majority recalculates over the regions that still speak. Five regions where one goes dark becomes a vote of four.

A silent region is not a down vote and not an up vote. It leaves the vote until it reports again.

I think this is the only honest reading. A region that cannot reach the brain is telling you nothing about your site. Treating its silence as "down" would turn every probe outage on the monitoring side into a fake incident on yours.

And it works in both directions. Missing data never opens an incident, and it never closes one. An open incident only closes when the down votes fall below the threshold and at least one region shows a real run of passing checks. Recovery needs proof, the same way failure does.

Who watches the watchers

There is one gap left. If every region that covers your monitor goes dark, the vote has nobody in it. Your site could burn down and no incident would open, because no data means no votes.

For that case there is a separate signal, one level below the checks. Every probe sends a small check-in to the brain on its own schedule, a heartbeat that has nothing to do with your monitors. When the last live probe covering a monitor goes stale, you get a different message: "NO DATA: monitoring interrupted, no check results received." It is honest about what it knows: the service cannot see your site right now. It does not claim your site is down, because it has no idea.

When probing returns, you get a "monitoring RESUMED, receiving check results again" note, and the vote picks up where it left off.

One more detail I like. If a large share of all monitors goes silent at the same time, that pattern is almost never a thousand customer problems. It is one problem, mine. In that case the system alarms me and holds the customer notices, so my bad day does not become spam on yours.

What this means for your setup

Three practical points follow from this, whatever tool you run.

Check from more than one region. The vote cannot protect you with a single location. Two regions with the majority rule means both have to agree, which already kills the classic false alert. Three gives you two out of three, which is the right balance for most services.

Leave the confirmation count at two unless you have a reason. It is the difference between "a packet got lost" and "this endpoint is failing."

Pick the rule to match the monitor. A payment API deserves majority or even any. An internal tool nobody uses at night can wait for all. The setting is per monitor, so you do not have to choose one policy for everything.

The goal of all this machinery is boring: when your phone buzzes, it is real. Everything else is plumbing.

Common questions

What is a false alert in uptime monitoring?

A false alert says your site is down when it is not. The usual cause is a problem near the probe, not near your site: a bad network path, a busy datacenter, a DNS hiccup on the monitoring side. Your site answered fine for every real user the whole time.

How many regions need to agree before an incident opens?

By default, more than half of the regions that are reporting results. A region only joins the down side after it fails the same check twice in a row. You can change the rule per monitor: any single region, all regions, or a fixed count.

What happens when a probe region goes offline?

It leaves the vote. A region that sends no results is not counted as down and not counted as up. The vote recalculates over the regions that still report. Missing data alone never opens an incident.

Does a silent region close my open incident?

No. Closing needs real proof: the down votes must fall below the threshold and at least one region must show a run of passing checks. Silence is not proof of recovery, so an open incident stays open until real results come back.

Key takeaways

One failed check from one place proves nothing. Confirm it from the same region before that region gets a say.

Give each confirmed region one vote, then require agreement. A majority of reporting regions is a good default.

A silent region must leave the vote, not join the down side. Otherwise every probe outage becomes a fake incident.

Missing data must not close an incident either. Recovery needs a real run of passing checks.

Cover the all-dark case with a separate heartbeat and a message that says "we cannot see your site," not "your site is down."

Two regions kill the classic false alert. Three is the sweet spot for most services.

The three figures above are live widgets in the original post: you can change the confirmation count and the vote rule and watch where the line falls. The whole project is open source, so the scheduler and the incident writer are there to read.

How does your setup handle a probe that goes quiet? I have seen tools count silence as down, and I would like to know if anyone has a better reading than "it leaves the vote."

How I mapped my codebase for humans and AI agents

Slim — Thu, 23 Jul 2026 14:17:23 +0000

TL;DR. I asked an AI model to turn my codebase into three things: a one-page summary for me, a JSON file for the next AI agent, and an interactive map you can click. It worked well, but only after one boring step: check every number against the code. The model got the shape right and several counts wrong. You can see the live map.

The interactive map itself, with one flow lit up across the system. Open the live version and click any flow.

A big codebase does not fit in your head. Mine is about 146,000 lines of Rust across 31 modules. When you open a project that size, the first hour is just finding where things are.

AI agents have the same problem. Every time an agent starts a task, it reads many files to learn how the system fits together. It does this from zero, every time. That is slow, and it costs money.

So I tried something. I asked an AI model to read the whole codebase and write three things. One for me, one for the next agent, and one for anyone.

The three files

The first file is a one-page summary for a human. It lists the rules that must always hold, the main parts and what each one does, the path a request takes, and the traps that waste time. It is the page I wish existed on my first day.

The second file is a JSON map for the next AI agent. It is not written to be pretty. It lists the invariants with the file or test that enforces each one, a short recipe for each common task, the known traps, and the key files. When the next agent starts a task, it reads this first and skips an hour of searching.

The third file is an interactive map for everyone. It shows the parts as boxes in columns. You pick a flow, like "a scheduled check" or "an agent login", and the path lights up across the boxes with numbered steps. It is live here: the architecture map.

The three files. The JSON map is the source of truth, and the human page and the interactive map are both built from it.

One file is the source of truth

The JSON map holds every fact. The other two files are only views of it. Build the map first, and the human page and the interactive map cannot disagree.

The method, not one magic prompt

There is no single prompt that does this well. The result comes from four steps, in order.

The four steps. Everything hangs off step two, the map.

First, explore in parallel. One agent cannot read 146,000 lines in one go. I let several agents read different parts at the same time, then joined their notes. This is faster and it covers more.

Second, build the machine map first. The JSON for the agent is the source of truth. The human page and the interactive map are only views of it. So I build the JSON first and put every fact in one place.

Third, check every number in that map, before you build anything else. This is the step people skip, and it is the most important one. You verify once, at the source, so a wrong count cannot spread into the other files.

Fourth, render the views from the map. The human page and the interactive map both come from the same JSON, so they cannot disagree. Each view has one clear reader: the page is for a new engineer, and the map is for a visitor who has never seen the code.

The numbers the AI got wrong

The model was good at structure. It found the parts, the flows, and the rules. But it guessed numbers, and some guesses were wrong.

It said there were 13 alert channels. The real number is 14.

It said there were about 80 error codes. The real number is 155.

It said there were 19 blog posts. The real number is 18.

None of these are small. If you publish them, you look careless, and the next agent trusts a wrong map. I only found them because I checked each count against the source code.

Check the numbers, at the source

An AI model is strong at shape and words. It is weak at exact numbers. Verify every count in the map, once, before you build anything from it. Then check the ones that matter yourself.

The prompts

Here are the three prompts, in the order I ran them. Build the map first, check it, then make the views. Change the details for your own project, and keep the "verify against the source" line in the first one.

For the machine map, build this first:

Read my whole codebase. Write one JSON file for the next AI agent that
will add a feature. Include the invariants, and for each one name the file
or test that enforces it. Add a short recipe for each common task: the
goal, then the files to touch in order. Add the known traps and the key
files. Keep every path exact. Verify every count against the source code.
Do not invent numbers. This file is data for a machine, not prose.

For the human summary, built from the map:

From that verified JSON, write one self-contained HTML page that explains
the system to a new engineer. Include the rules that must always hold,
the main parts and what each does, the path a request takes, the path data
takes, and the traps that waste time. Rank the parts by size. Do not add
any number that is not already in the JSON.

For the interactive map, built from the same map:

From the same JSON, build one self-contained interactive HTML page. Show
the parts as boxes in columns. Show each flow as a numbered path that lights
up across the boxes when I click it. Keep all styles and scripts in the
page. No build step.

Why this is worth doing

The human page saved me time the next week. The map helps me explain the system in one screen. And the JSON is the part I did not expect to like. The next agent that touches this code reads a map first, so it starts from step one instead of step zero.

The next agent starts from step one, not step zero

A map file hands the next AI agent the rules, the tasks, and the traps up front. It reads one small file instead of re-reading the whole codebase every time.

Key takeaways

Build the machine map (JSON) first. The human page and the interactive map are only views of it, so they cannot drift.

An AI model is strong at structure and weak at numbers. Verify every count at the source before you build anything from it.

Write a map file for the next AI agent, so it starts from step one instead of re-reading the whole codebase.

Explore in parallel. Several agents reading different parts cover more than one agent reading everything.

Give each file one clear reader. It keeps the writing simple.

The model got real counts wrong, like 13 channels when the real number was 14. Do not publish a number you did not check.

If you want to see the output, open the interactive map. The whole project is open source, so you can read the real code behind every box.

If you have mapped a codebase for an AI agent: which format did the agent actually use, and how many numbers did the model get wrong on your run? Mine missed three. I would like to know if that is typical.

How to write incident status updates that build trust

Slim — Tue, 21 Jul 2026 13:42:58 +0000

TL;DR. During an outage your status page has a second job: holding trust while the fix is still in progress. Write plainly. Say what customers feel first, say what you are doing, and promise a time for the next update. Then keep that promise, even when there is no news. The four stages are investigating, identified, monitoring, and resolved.

The idea of an incident commander did not come from software. It came from wildfires. In 1970s California, a run of huge fires pushed fire chiefs to study why large responses fell apart. The cause surprised them. It was not too few firefighters or too little water. It was communication. Different teams used different words and no one shared a plan. The fix was a simple system with clear roles and clear language, now known as the Incident Command System, and on-call engineers still use a version of it today.

Your status page is the communication part of that system. When something breaks, the repair happens in your code. The trust happens on the status page. This post is about the second part: how to write updates that keep people calm while the first part is still in progress.

The words cost real money

An outage is expensive before you write a single word. In ITIC's 2024 survey of more than 1,000 companies, over 90% of mid-size and large firms said one hour of downtime costs them more than $300,000. For 41% of them, one hour costs between $1 million and $5 million. Uptime Institute's outage research points the same way: about one in five recent outages cost more than $1 million, and more than half cost over $100,000.

You do not need to be a bank for this to matter. If your store makes $2,000 a day, one bad hour during a sale can cost more than a slow afternoon, because sales are not spread evenly across the day. The size of the number changes. The shape of the problem does not.

Here is the part teams forget. Some of that cost is the downtime itself. The rest is trust, and trust is where writing helps. Many providers offer an SLA, a promise to stay up with a penalty if they miss it. The penalty is almost always a service credit, which is money back on your bill. A credit refunds your invoice, not your customer's bad afternoon. Good updates cannot bring the service back. They can protect the relationship while it is down.

The four stages of an incident

Most status pages, including the big public ones, use the same four stages. They come from a shared standard, so a customer who has read one status page already understands yours.

Stage	What it means	What to write
Investigating	You know something is wrong. You do not know why yet.	Name the impact and say you are looking into it.
Identified	You found the cause and are working on a fix.	Say what is wrong in plain words and that a fix is coming.
Monitoring	The fix is live. You are watching to be sure it holds.	Say the fix is in and you are checking that it works.
Resolved	The service is back and stable.	Say it is over, and thank people for waiting.

The order matters. Do not jump to identified because you have a hunch. If you say you found the cause and then change your story an hour later, every update after that is worth less. Move a stage forward only when the facts move with you.

The rules of a good update

Five rules cover almost every message.

Start with impact. Say what the customer feels before you talk about servers or databases. "Some payments are failing" is more useful than "we are seeing elevated 5xx errors on the API gateway." Only the first one means anything to a customer.

Share facts, not guesses. Say what you have confirmed. If something is still unknown, say that too. Writing "we do not yet know the cause" is fine. Inventing a cause to sound in control is not. This rule earns its place under pressure: the Uptime Institute's 2025 report found that about 40% of organizations had a major outage caused by human error in the last three years, and 85% of those came from someone not following a procedure. An incident is exactly when tired people guess.

Promise the next update. Give a real time, like "next update by 15:00 UTC" or "another update within 30 minutes." This one line does more than any apology. It tells people they can close the tab and get on with their day.

Add a timestamp and a time zone. "In 30 minutes" means nothing if the reader does not know when you posted. Every update should carry a clear time.

Keep it plain. Short sentences. No blame, no jargon, no backstory. A worried customer reads fast.

A bad update and a better one

Here is a bad update of the kind you have all seen:

We are aware of an issue and our team is working hard to resolve it as quickly as possible. We apologize for any inconvenience.

It sounds polite and says nothing. What is broken? Who is affected? When will I hear more? A better version answers those questions:

Investigating: some customers cannot log in. Sign-ups and password resets are affected too. Our team is looking into the cause now. Next update by 14:30 UTC.

Same length. Only one of them lets a customer decide what to do next.

How often to post

Set the rhythm in your first message and keep it. Every 30 to 60 minutes is a good starting point for a serious outage. The exact number matters less than the promise. If you said 30 minutes, post at 30 minutes, even when the only news is "still working, no change yet." Silence after a promised time reads as "they have lost control." A boring on-time update beats an exciting late one.

Planned maintenance is the same skill, calmer

Maintenance uses three stages: scheduled, in progress, and completed. The difference is that you have time to write it well in advance. Tell people what will happen, when it will happen, and whether they need to do anything. Mark it in progress while the work runs, and completed when it is done. The tone is calmer because nothing is on fire, but the shape is the same: impact, timing, next update.

The mistake almost everyone makes

It is not the wording. It is the second update.

The first update is easy. The outage just happened, and everyone is paying attention. Then the fix takes longer than expected, the team goes quiet, and the page sits at "investigating" for two hours. From the outside, a frozen status page looks the same as a dead company.

The fix is a habit, not a talent. Decide the next-update time in your first message, set a timer, and post again when it rings. If there is no news, that is the update: no change, still working, next check in 30 minutes. The point of the promise is that people stop refreshing and trust you to come back.

Write the message, then give it a home

You can write all of this by hand during an outage. Most people write worse under stress, which is the worst possible time to start from a blank box. That is why we built a free incident update generator: pick a stage, fill in a few fields, and it writes a clean message you can paste into any status page. It runs in your browser and stores nothing you type.

The message still needs somewhere to live. A status page is where customers look when your product will not load, so it should be honest and easy to reach. If you are choosing an uptime number to promise, the uptime SLA calculator turns a percentage into real minutes, and is 98% uptime good shows how small percentages hide large amounts of downtime. For the page itself, a status page you cannot fake covers how to keep it worth trusting.

Common questions

What should the first incident update say? Say what the customer feels, say you are looking into it, and give a time for the next update. You do not need the cause yet. Waiting for the root cause before posting is the most common mistake.

What is the difference between investigating and identified? Investigating means the cause is not confirmed. Identified means you have confirmed the cause and are working on a fix. Do not move to identified based on a hunch.

Should you apologize in an incident update? One short, honest line is fine. Skip the long apology. A clear next-update time does more for trust than three sentences of sorry.

If you have run incidents: what update cadence actually held up for you under a long outage, and did anyone ever complain that you posted too often? I have never seen that complaint, but I would like to know if it exists.

Why I chose Rust over Go for an uptime monitor

Slim — Mon, 20 Jul 2026 14:46:30 +0000

I build Uptimepage, an open-source uptime monitor and status page written in Rust. People ask why Rust and not Go, since Go is the usual pick for this kind of network service. Here is the honest answer. It is not that Rust wins everywhere. It is that one part of this job made the choice for me.

The product is one promise: tell you fast and honestly when your site is slow or down. The numbers I show you, like your p99 response time, have to be clean. If my own code adds random delay, I blur the exact signal you pay for. So the runtime under the prober matters more here than it would for a normal app.

Garbage collection shows up in the tail

Go has a garbage collector. It is fast, and most apps never feel it. But it still has to do work to free memory, and that work can land inside the millisecond timings I report. Run tens of thousands of checks at once and a pause at the wrong moment lifts a p99 number. At that point I am measuring my own runtime, not your server.

Rust has no garbage collector. Memory is freed at a point I can see in the code. There is no background pause I did not write. For a tool that sells timing, that control is worth the extra work Rust asks for.

What that looks like in memory

Here are some real numbers, with a warning attached. They come from a load test on a developer laptop, not a production server. I use them to catch a slowdown between two versions of my code, not to plan capacity. A real server does better, a small box does worse. Treat them as a floor.

In one run, a single machine held 50,000 checks in flight and peaked at 933 MiB of memory. That is under one gigabyte for fifty thousand live checks. My running server uses about 42 MiB while watching its monitors, and it sits quiet when there is nothing to do. That kind of density is normal for a service with no garbage collector, and it means one small box covers a lot of monitors. I go deeper on the prober and the throughput numbers in the build story.

The bug that never compiles

Speed is only half of it. The other half is a class of bug that Rust refuses to build.

Picture many workers writing to one shared map of results at the same time. In Go this compiles and runs. Sometimes it is fine. Sometimes two goroutines write at once, you get corrupt data or a crash, and it only happens under load, which is the worst time to find out:

// compiles fine, races at runtime
results := map[string]int{}
for _, check := range checks {
    go func(c Check) {
        results[c.ID] = c.Latency // data race
    }(check)
}

Go gives you good tools for this. There is a race detector, a sync.Mutex, and channels. But remembering to reach for them is on you.

In Rust the same concurrent write does not compile. The compiler stops you until the shared map is wrapped in a lock:

// will not compile unless the shared map is a Mutex
let results = Mutex::new(HashMap::new());
std::thread::scope(|s| {
    for check in checks {
        let results = &results;
        s.spawn(move || {
            // each worker locks only for its own insert
            results.lock().unwrap().insert(check.id, check.latency);
        });
    }
});

Each worker holds the lock only for its own insert, so the writes stay safe without blocking the others for long. For code that runs day and night across many machines, "the compiler will not let you ship the race" removes a whole set of late-night bugs before they exist.

Where Go is the better pick

None of this makes Go a bad choice. Often it is the better one. Go is faster to learn. It builds in seconds. Its standard library for network services is excellent, and a new engineer can be useful in days. If I were building a normal web service, or something I had to ship this week, Go would be on the table and might win.

Rust asks more from you first. The compiler argues with you. The build is slower. You spend time on things Go would just handle. I take that trade because this job is narrow and it rewards tight control over memory and timing.

The compiler as a safety net

Heavy code review is how large teams catch data races and memory bugs. Rust gives you a lot of that for free. The compiler turns those mistakes into build errors, so they never reach production and never wake anyone up. Every change gets a strong, automatic check before it ships, which is a big part of why I trust the service to run unattended.

It is also why Uptimepage is open source and self-hostable. You can read the code, run it on your own hardware, and export your data whenever you want, so you are never locked into a single vendor. You can start from GitHub.

The honest version

The honest version is not "Rust beats Go." It is "for a tool that lives or dies by timing and runs at high concurrency, Rust fit better." Pick the language for the job in front of you. Mine happened to be a job that Rust is very good at.

If you have shipped a heavily concurrent network service, did you reach for Go or Rust, and did the GC ever actually show up in your tail latencies? Curious where people land when the timing is the product.

Is 98% uptime good? It allows 7.3 days of downtime a year

Slim — Fri, 17 Jul 2026 13:26:37 +0000

Cover photo by the blowup on Unsplash.

I build an uptime monitor, so this question lands in my inbox a lot: is 98% uptime good?

For a public website or a paid API, no. For an internal tool or a side project, it is fine. The difference is one division away.

98% looks like a top grade because school taught us that 98 out of 100 is excellent. Uptime does not grade like school. The whole scale for public services lives between 99% and 100%, and serious targets differ only in the digits after the decimal. On that scale, 98% sits at the bottom.

Turn the percentage into time

The allowed failure at 98% is 2%. Two percent of a year is 7.3 days. Two percent of a 30-day month is 14.4 hours. If your shop makes $2,000 a day, 98% uptime means you accept about $14,600 of closed-door time per year and the target still counts as met.

Here is the same math for every common target. The last column prices the downtime for that $2,000-a-day shop.

Uptime	Per day	Per 30-day month	Per year	Lost sales per year
90%	2.4 hours	3 days	36.5 days	$73,000
95%	1.2 hours	36 hours	18.3 days	$36,500
98%	28.8 minutes	14.4 hours	7.3 days	$14,600
99%	14.4 minutes	7.2 hours	3.7 days	$7,300
99.5%	7.2 minutes	3.6 hours	1.8 days	$3,650
99.9%	1.4 minutes	43 minutes	8.8 hours	$730
99.95%	43 seconds	21.6 minutes	4.4 hours	$365
99.99%	8.6 seconds	4.3 minutes	52.6 minutes	$73
99.999%	0.9 seconds	26 seconds	5.3 minutes	$7

Two jumps on this table do most of the work in real contracts. From 98% to 99.9%, the allowed downtime drops from 14.4 hours a month to 43 minutes. From 99.9% to 99.99%, it drops from 43 minutes to 4.3 minutes, and that second jump usually costs about ten times more engineering than the first while saving the shop $657 a year.

The shape of the downtime matters

98% per month is 14.4 hours, but the number says nothing about how those hours land.

Thirty minutes of planned maintenance every night at 03:00 adds up to 98% and most users never notice. One 14-hour outage on the day of your product launch is also 98%. Same score, very different month.

So a single uptime percentage is a summary, not the full story. When someone quotes you a number, also ask about the longest single outage and when it happened.

When 98% is enough

Plenty of systems can live at 98% and nobody gets hurt:

An internal wiki. People retry after lunch.
A staging environment. Downtime there is often planned.
A batch job that builds reports at night. It has hours of slack before anyone reads the output.
A home server on a residential connection. Your power company already decided your uptime for you.

The shared pattern: when these go down, nobody loses money and nobody loses trust. Paying for more nines there is waste.

When it is not

A checkout page, a paid API, a login service. Here 98% fails twice. First the direct cost: 14.4 hours a month of failed requests and support tickets. Second the trust cost, which is larger and slower. A customer who hits your outage twice in one week does not check your uptime report. They remember that your service is the one that breaks.

One more trap: an SLA is not uptime. Uptime is the measured number. An SLA is a contract promise with a penalty, and the penalty is almost always a service credit. If a provider misses its 99.9% SLA, you get part of your bill back. Your customers get nothing.

What to aim for instead

99.9% is the default target for customer-facing services for a reason. 43 minutes a month is enough room for a bad deploy and a couple of small failures, and a small team can hit it without heroics. Above that, each nine costs roughly ten times more and most users cannot feel the difference.

If you want to run your own numbers, I keep a free uptime SLA calculator and an error budget calculator on our site; both work without signup.

What target do you actually run in production, and did you pick it or inherit it? I am curious how many teams measured before they promised.

The status page you can't fake: measured uptime, not published

Slim — Wed, 15 Jul 2026 13:50:26 +0000

A status page is the one dashboard a company publishes about its own service. It is also the one place where the company has a reason to look good. That is a problem. If the page can be edited to look better than reality, it stops being useful. So when you build or choose a status page, ask one thing: can someone hide a real outage on it?

The short answer: a status page you can trust builds its uptime bar from real checks, not from the incidents a person chose to publish.

Two layers, and only one is yours to edit

A good status page has two parts. The first part is the measured data: the green and red timeline, the 90-day bar, the uptime percent. The second part is incidents: the notes a person writes to explain what broke, what they are doing, and when it is fixed. On the page they sit next to each other and look the same. They are not the same, and mixing them is where trust gets lost.

The measured data answers one question: what did the checks see? The incident notes answer a different one: what does the team want to say about it? The first is a fact. The second is a story. A status page you can trust lets the team write the story, but keeps them away from the facts.

The story layer also includes the postmortem, the write-up you post after an outage. A postmortem is honesty you add on purpose. You explain what broke and why, because you choose to. The bar works the other way. It shows the failure on its own, whether you write anything or not. You control the story. You do not control the facts.

The trap: a bar that only shows what you published

The common mistake is to build the uptime bar from incidents. It feels natural. You already open an incident when something breaks, so why not color the timeline from incidents too? Now the bar turns red only where an incident exists and is marked public.

The problem comes on the day an outage has no public incident. Maybe no one published it. Maybe a setting was wrong. Maybe the monitor was added to the page after the outage, so its incident was saved as private and never checked again. In every case the timeline shows green over a real red day, and it does this quietly. The uptime number goes up. The customer sees 100 percent over a week they remember as broken.

One version of this trap is easy to build by accident, so it is worth explaining. You decide "is this incident public?" once, at the moment the incident opens, based on whether the monitor was on a public page right then. Then you save that answer and never look again. In the code it looks like a live check. It is really a photo taken one time. Move the monitor onto the page a day later, and its past outages stay hidden, because the photo was taken before the monitor was there. Any yes or no mark that is set once and then trusted forever has this problem.

The fix is a rule, not a switch

The bar should come from the measured data. It needs only one rule to stay calm: wait for confirmation before you count downtime. You do not want one failed check from one place to turn a whole day red, because networks are noisy and one bad checker is not an outage. So you wait for agreement: more than one region failing at the same time, for more than one check in a row. That one rule is enough to keep a short blip from becoming a red day.

One thing matters here: the same rule feeds both your alerts and your uptime bar. If the rule that wakes your on-call person is the same rule that colors the timeline, the two can never tell different stories. The moment you add a second rule just for the bar, it will drift from the first, and the page will disagree with itself. On-call gets paged, but the public history says everything was fine.

Publishing stays where it belongs, on the notes. The team decides whether to write an incident, what to say, and when to post the all-clear. They do not decide whether last Tuesday was down. The checks already decided that.

Test the status page you already have

You can check any status page, including your own, in a few minutes.

Add a monitor to a page after it has already had an outage. Does the history show the outage, or does it start clean from the day you added the monitor?
Take a real incident and unpublish it. Does the bar keep the red day, or does the day turn green?
Make one region fail for one second. Does the whole day go red, or does the bar stay calm?

A page you can trust shows the outage in the first test, keeps the red in the second, and stays calm in the third. A page that fails these is usually not lying on purpose. It just built its timeline on top of what people chose to publish, and that always has holes.

Where this leaves you

This is how I built it into Uptimepage: the 90-day bar and each status light come from confirmed downtime, measured across regions with a confirmation rule, not from what someone chose to publish. Incidents and postmortems are the layer you write by hand. It is AGPL and open source on GitHub, and there is a longer write-up on the internals if you want the Rust side.

You should not be able to fake your uptime. You should not be able to fake it by accident either. The bar is a measurement. Keep it one.

How does your status page compute its uptime bar, from checks or from published incidents? Curious what people run and whether anyone has been bitten by the frozen-flag version of this.

Error budgets, explained: SLOs, burn rate, and when to stop shipping

Slim — Mon, 13 Jul 2026 14:11:33 +0000

The idea

An uptime target has a second number hidden inside it. Say you promise 99.9% uptime. You are also saying that 0.1% is allowed to fail. Put that 0.1% into real time, and that is your error budget: the downtime you can have before you break the promise.

This changes how you look at downtime. It stops being a mistake to feel bad about and becomes a budget you can spend: on a risky deploy, on a slow service you depend on, or on a migration. When the budget runs low you slow down. When it is healthy you can move fast.

The formula

An SLO is your reliability target, for example 99.9%. The gap to 100% is the failure you are allowed. Multiply that gap by the length of the window, and you get real time you can spend.

error budget = window x (1 - SLO)

A 99.9% SLO over a 30-day month is 2,592,000 seconds times 0.001. That is 43 minutes and 12 seconds. That is the whole budget for the month, shared across every incident, not a fresh 43 minutes each day.

So three nines is not "never go down". It is a 43-minute budget each month. Every extra nine costs about ten times more engineering, for downtime that most users never notice. This is why Google says that 100% is the wrong target for almost every service.

Burn rate is the speed

The total tells you how much you can spend. It does not tell you how fast. Two services can both sit at 99.9% for the month: one loses the budget slowly, the other loses it all in a single bad hour. Burn rate tells them apart.

burn rate = (1 - measured) / (1 - SLO)

A burn rate of 1x spends the whole window exactly. 2x spends it in half the time. Below 1x, you finish the month with budget left. Above 1x, the rate tells you the deadline:

budget runs out in = window / burn rate

At 2x, a 30-day budget is gone in fifteen days. At 14.4x it is gone in about two days. That same 14.4x spends 2% of the budget in one hour, which is the level most fast alerts are set to.

Turning burn rate into alerts

One threshold is not enough. It either alerts too late or it sends too many false alarms. The common fix uses two windows: a long one to confirm the problem is real, and a short one to clear the alert quickly once you fix it. Both have to be burning for the alert to fire. For a 30-day budget, these are the usual settings:

Fast page: 2% of the budget in 1 hour (with a 5-minute short window). This is a 14.4x burn.
Page: 5% in 6 hours (30-minute short window). This is a 6x burn.
Slow ticket: 10% in 3 days (6-hour short window). This is a 1x burn.

The fast page catches a sudden outage. The slow ticket catches a slow problem that would still use up the whole month if nobody looked.

The rule is the point

The math is the easy part. The value comes from a rule you agree on before anything breaks. The rule is simple. When the budget runs out, risky launches stop, and the team works on reliability until the budget grows back. While the budget is healthy, you ship and you take the risk.

One more rule keeps it fair. If you never spend your budget, your SLO is too strict, and you are paying for reliability that no user asked for.

Try it

I put the formulas into a free, no sign-up error budget calculator: enter an SLO and your measured availability, and it shows budget spent, budget left, burn rate, and a burn-down chart. There is also an uptime SLA calculator for the full downtime-per-nine table.

What SLO and burn-rate thresholds do you run in production? Curious how others pick them.

ClickHouse system tables ate my disk (and the fix)

Slim — Thu, 09 Jul 2026 15:13:13 +0000

TL;DR.

A monitoring alert said I was dropping check results. The disk was 100% full. My actual data was 20 MB. ClickHouse had quietly written 12 GB of logs about itself, mostly system.text_log and system.trace_log. Those same logs also burn CPU at idle. The fix is a few lines of ClickHouse config that disable the noisy logs and slow the metrics collector. Full config is below.

The alert

One morning a critical alert fired: UptimepageResultsLost. Its description is blunt: storage write failures or dropped results, checks run but results are not persisting. Then it did something worse than fire once. It cleared, fired again, cleared, and fired again, over and over.

I run Uptimepage, an uptime monitoring service. "Results not persisting" means the one thing customers pay for, recording whether their sites are up, might be failing. So it had my full attention.

The good news first: no data was lost. The write path retries, and every failed write was caught by a retry. I keep a counter for results that actually get dropped, and it stayed at zero the whole time. But something was clearly wrong underneath.

The real signal: a full disk

The first real signal came from Postgres, which had crashed and restarted a couple of minutes earlier:

FATAL: could not write lock file "postmaster.pid": No space left on device

The disk was 100% full. Postgres could not write its lock file, crashed, and recovered on its own through WAL replay. ClickHouse, where I store raw check results, was rejecting inserts with its own version of the same complaint:

Code: 243. DB::Exception: Cannot reserve 1.00 MiB, not enough space:
While executing WaitForAsyncInsert. (NOT_ENOUGH_SPACE)

So the alert was a symptom. The real problem was a full disk. That reframes the question: I do not store much, so what filled it?

Leak one: old Docker images

docker system df gave the first half of the answer:

88 Docker images, only two of them in use. Every deploy pulls a fresh image, and nothing pruned the old ones, so they piled up for weeks. A docker image prune -af reclaimed about 10 GB and took the disk off the ceiling.

That stopped the bleeding. But 13.8 GB of volumes is a lot for a service whose data I thought was tiny. That number turned out to be the real story.

Leak two: ClickHouse logging about itself

I went into ClickHouse and asked the obvious question, which table is big:

SELECT database, formatReadableSize(sum(bytes_on_disk)) AS size, sum(rows) AS rows
FROM system.parts
WHERE active
GROUP BY database
ORDER BY sum(bytes_on_disk) DESC

The answer stopped me:

Database	Size	Rows
`system`	11.8 GiB	577,340,927
`monitor`	19.7 MiB	1,813,884

monitor is my data: every check result and rollup I keep. About 20 MB. The system database, ClickHouse's own diagnostic tables, was 11.8 GiB. Nearly all of the storage was ClickHouse logging about itself.

Breaking system down by table:

Two tables did most of the damage. system.text_log (5.29 GiB) is a copy of the server's own log output written into a table. system.trace_log (3.02 GiB) is the query profiler, which samples running queries. Both are handy when you are actively debugging ClickHouse. Neither is worth multiple gigabytes when I am not. And text_log is off by default, so something in my setup had switched it on.

The fix

Two parts: reclaim the space now, and stop it coming back.

Reclaim now. The system log tables are throwaway diagnostics, not real data. Truncate them:

TRUNCATE TABLE IF EXISTS system.text_log;
TRUNCATE TABLE IF EXISTS system.trace_log;
-- and the rest of the system.*_log tables

That gave back 12 GB at once and took the disk from 74% down to 41%.

Stop the regrowth. Truncating is a one-time cleanup. Without a cap, the logs fill right back up. The durable fix is ClickHouse config, added under /etc/clickhouse-server/config.d/. I watch ClickHouse through Grafana, not these tables. So I disable almost all of them and keep only query_log and part_log, both bounded, for the rare hands-on debugging session:

<clickhouse>
    <!-- Sample async metrics every 60s, not every second. -->
    <asynchronous_metrics_update_period_s>60</asynchronous_metrics_update_period_s>

    <!-- Disable the log tables. remove="1" on an absent table is a no-op,
         so this list is safe to paste as-is across ClickHouse versions. -->
    <asynchronous_metric_log remove="1"/>
    <asynchronous_insert_log remove="1"/>
    <backup_log remove="1"/>
    <error_log remove="1"/>
    <crash_log remove="1"/>
    <metric_log remove="1"/>
    <query_metric_log remove="1"/>
    <query_thread_log remove="1"/>
    <query_views_log remove="1"/>
    <session_log remove="1"/>
    <text_log remove="1"/>
    <trace_log remove="1"/>
    <opentelemetry_span_log remove="1"/>
    <zookeeper_log remove="1"/>
    <processors_profile_log remove="1"/>
    <latency_log remove="1"/>
    <background_schedule_pool_log remove="1"/>

    <!-- Keep query_log and part_log, bounded, for on-hand debugging. -->
    <query_log>
        <ttl>event_date + INTERVAL 3 DAY DELETE</ttl>
        <max_size_rows>1048576</max_size_rows>
    </query_log>
    <part_log>
        <ttl>event_date + INTERVAL 3 DAY DELETE</ttl>
        <max_size_rows>1048576</max_size_rows>
    </part_log>
</clickhouse>

remove="1" disables a system log completely. The <ttl> element on query_log and part_log adds a TTL and keeps the table's default partitioning, so you do not have to restate the whole engine. ClickHouse picks up the change after a restart. Altinity's "system tables ate my disk" note covers the same ground and is worth a read. If you would rather keep the diagnostics, give every log a short <ttl> instead of disabling it.

Bonus: lower CPU, not just disk

Those log tables are written constantly, and on a small server that steady write load shows up as CPU. A long-running ClickHouse issue tracks the server using noticeable CPU at zero load, #60016. People there report dropping from 40 to 70% CPU down to about 1.5% after disabling the logs and slowing the async-metrics collector. So those two settings, the asynchronous_metrics_update_period_s line and the remove="1" block, pay off twice: less disk and less CPU.

The real lesson: alert on the cause, not the effect

Here is the part that stings. I had an alert for "results are being dropped." I did not have an alert for "the disk is filling up." So a full disk is a slow, predictable problem that builds over days. But it only reached me as a sudden downstream symptom, after Postgres had already crashed once.

A downstream alert like "results lost" is not a substitute for watching the resource that actually runs out. I added the missing one: a plain host disk-space alert on node_filesystem_avail_bytes, firing at 80% and 90% used, well before anything starts failing. That is the alert that should have caught this on day one.

Key takeaways

ClickHouse system log tables are unbounded by default and can dwarf your real data. Mine were 11.8 GB against 20 MB of actual data.

system.text_log and system.trace_log are the usual offenders. text_log is off by default, so check whether something enabled it.

Cap them in config: remove="1" to disable a log, or <ttl> plus <max_size_rows> to bound the ones you keep. Truncate to reclaim space right away.

It is not just disk. The same logs burn CPU at idle on small servers. Disabling them, plus asynchronous_metrics_update_period_s = 60, took reporters in ClickHouse issue #60016 from 40 to 70% CPU down to about 1.5%.

Anything that pulls artifacts on a schedule, Docker images in my case, needs matching cleanup or it becomes a slow disk leak.

Alert on the cause (disk space), not only the effect (dropped writes). The cause gives you days of warning; the effect gives you minutes.

This happened on Uptimepage, the uptime monitor I run and dogfood.

Postgres vs ClickHouse? I use both. 4 tricks from the split.

Slim — Thu, 09 Jul 2026 05:55:03 +0000

TL;DR. My uptime monitor keeps config and incidents in Postgres and check results in ClickHouse. The split is one rule: does a row ever change? Config gets edited, so it wants Postgres transactions and constraints. A check result is written once and never touched again, so it goes to ClickHouse, where the right codec, a careful sort key, and a per-row TTL make billions of rows cheap. Four tricks and a bonus below, useful even if you only ever run one database.

Every monitor I run writes a row every 20 seconds, from every region, and never stops. One monitor is about 4,300 rows a day, per region. Multiply by every monitor on the platform and the rows only ever go up.

That stream would slowly crush a normal Postgres table, and watching it is the whole product. So the check results do not live in Postgres. They live in ClickHouse. Everything else, the monitors and incidents and teams, lives in Postgres. The interesting part is the line between them.

People ask why not one database. "Postgres vs ClickHouse" is the wrong question, because the two are not fighting over the same job. Here is the line I draw, and the one rule under all of it: split your data by how it is written, not by which engine is faster.

1. Does the row change? That picks the database

Forget benchmarks for a second. One question sorts a table into one store or the other: after you write a row, will you ever change it?

one monitor, two stores:

  you edit a monitor  ->  Postgres   (targets, incidents, team)
                          the row changes, has constraints, lives in a transaction

  a probe checks it   ->  ClickHouse (check_results)
                          one row, written once, never updated

A monitor is a row that changes. You toggle it off, edit the URL, change the interval. So it lives in Postgres:

CREATE TABLE targets (
    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name          TEXT NOT NULL,
    check_spec    JSONB NOT NULL,
    interval_secs INTEGER NOT NULL CHECK (interval_secs >= 10),
    enabled       BOOLEAN NOT NULL DEFAULT true,
    updated_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

That CHECK means a 3-second interval can never reach the table, no matter which part of the app tried to write it. This is Postgres doing the thing it is best at: a small set of rows that must stay correct while many callers change them at once.

A check result is the opposite. It is written once when a probe finishes, and then it never changes. Nothing ever updates it, so it does not need any of that:

CREATE TABLE check_results (
    org_id      UUID,
    target_id   UUID,
    region      LowCardinality(String),
    timestamp   DateTime('UTC') CODEC(DoubleDelta, ZSTD(1)),
    status      Enum8('up' = 1, 'down' = 2, 'degraded' = 3, 'error' = 4),
    duration_ms UInt32 CODEC(T64, ZSTD(1))
) ENGINE = MergeTree

No updated_at, no foreign key, no transaction. Just an append-only stream that grows forever. That is the shape ClickHouse is built for and the shape that slowly hurts Postgres.

Notice the types too. status is an Enum8, one byte on disk, not the string "up". region is LowCardinality, stored once in a dictionary and referenced by a small id instead of repeating the text on every row. On a table that only ever grows, a byte saved per row is a byte saved times billions.

2. The right codec turns a day of timestamps into almost nothing

Once the results are in a column store, the type on each column is most of the compression, and the default setting wastes a lot of space.

Look at that timestamp again. A monitor checks every 20 seconds, so for one monitor a day of timestamps is a run of 20, 20, 20, 20, over and over. DoubleDelta stores the change in the change. For a steady interval the first delta is a constant 20 and the second delta is zero, so each row after the first packs down to about a bit. The codec is not a ClickHouse invention: it comes from Facebook's Gorilla time-series paper, built for exactly this, measurements taken at a steady rate.

timestamp   DateTime('UTC')  CODEC(DoubleDelta, ZSTD(1)),
duration_ms UInt32           CODEC(T64, ZSTD(1)),
dns_ms      Nullable(UInt16) CODEC(T64, ZSTD(1)),

I also store the timestamp at whole seconds on purpose. The smallest interval is 20 seconds, so no two checks for one monitor ever land in the same second, and the sub-second detail lives in duration_ms where it belongs. Whole seconds that step by a fixed amount compress far harder than milliseconds would.

The latency columns get T64 instead, because they are small integers that stay in a small range, a different shape from a steady clock. T64 crops the unused high bits off a block of values, so a UInt32 that never climbs past a few thousand milliseconds stops paying for all 32 bits. That is the trick worth copying, and it is not the specific codec names. It is that a timestamp ticking by a fixed step and a latency staying in a small range are two different shapes, and telling the database which is which does the work.

3. Put the tenant in the sort key, not the partition

I got this one wrong first, so learn it from my mistake instead of your own.

This is a multi-tenant app, so every row carries an org_id and almost every query filters by it. My first schema partitioned by org, one slot per customer, because it felt tidy. ClickHouse turned that into a flood of tiny parts, the merges could not keep up, and startup got slower the more customers I had. I ripped it out. A partition is not a folder for tidiness. It is a physical unit ClickHouse merges and expires, and you want few large ones, not many small ones.

Here is what it should be:

) ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (org_id, target_id, region, timestamp)

org_id leads the ORDER BY, so a per-org query walks the sorted primary index and reads only that org's slice. But it stays out of PARTITION BY. The rule I follow now: partition by something low-cardinality that you also delete by, here the day, and put the high-cardinality tenant key in the sort order. Sort key answers "find this org fast". Partition answers "drop old data cheaply", which is the next trick.

4. Make retention a number in a row, not a schema change

Different plans keep history for different lengths of time. The clumsy way is a migration or a cleanup job per plan. The clean way is to make the retention window a column:

ttl_days UInt16 DEFAULT 30 CODEC(ZSTD(1))
) ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (org_id, target_id, region, timestamp)
TTL timestamp + toIntervalDay(ttl_days)

Each row carries its own ttl_days, stamped from the org's plan at the moment it is written. A free plan keeps 30 days, a paid plan keeps more, and changing that needs no ALTER, no migration, no backfill. The next write just stores a different number. The daily partitions line up with the TTL, so ClickHouse expires old data by dropping whole parts, close to free, and a whole column of the same 30 compresses away to nothing. The flexibility costs no space.

One more piece makes reading that history cheap. A materialized view rolls raw checks into per-minute and per-hour summaries as they land:

CREATE MATERIALIZED VIEW check_results_1m
ENGINE = AggregatingMergeTree
ORDER BY (org_id, target_id, region, minute)
AS SELECT
    org_id, target_id, region,
    toStartOfMinute(timestamp)  AS minute,
    countState()                AS total_checks,
    countIfState(status = 'up') AS up_checks,
    avgState(duration_ms)       AS avg_duration_ms
FROM check_results
GROUP BY org_id, target_id, region, minute;

So a dashboard showing 30 days across a thousand monitors reads a few thousand minute-buckets, not millions of raw rows. Recent views read the minute rollup, long history reads an hour rollup, and the raw rows underneath expire on their own TTL. The firehose is there when you need to drill into one bad minute, and left alone the rest of the time.

One more, because everyone hits it: make inserts idempotent

Here is the trick I wish someone had told me first. My agents batch check results and send them to ClickHouse. A network blip after the server commits but before my side gets the ack means the batch retries and sends the exact same block again. Without protection, that double-counts every row in it, and your uptime numbers quietly drift.

ClickHouse has a fix built in, but for a plain (non-Replicated) MergeTree it is off until you turn it on:

SETTINGS index_granularity = 8192,
         non_replicated_deduplication_window = 1000;

The server hashes each inserted block and remembers the last 1,000 hashes. A retry sends an identical block, the hash matches, and the server drops it instead of appending it again. The retry becomes safe to do blindly, so the client stays simple: send, and if unsure, send again. Make the window bigger than the most blocks a single retry could resend, and you stop having to worry about duplicate writes.

So, one database or two?

For almost every app, one. If your data fits in Postgres and stays fast, a second database is a cost you should not pay: two schemas, two clients, two things to back up and think about.

Reach for the second store only when one table stops looking like the rest of your tables. For me that table was check_results. It is written once and never edited, it grows without end, and every question I ask it is a summary over a time range. That is a different shape from my monitors and my incidents, so it wanted a different database.

The rule I would give my past self: do not split by "which database is faster". Split by how the data is written. Rows that change and must stay correct want Postgres. An append-only stream you only ever summarize wants a column store like ClickHouse. Most of the tricks above are that one idea pushed down into the schema.

Uptimepage is open source, AGPL-3.0, and both schemas are in the repo: github.com/uptimepage/uptimepage. The probe that writes those rows is its own post, and the wider build story, one binary and two databases, is here. Or start free on the hosted tier and point a check at something.