DEV Community: Remdore

92% of dev.to posts get zero reactions, and the title advice does not change that

Remdore — Wed, 16 Sep 2026 12:12:00 +0000

I set out to test the advice. Every few weeks someone publishes a piece explaining what works on dev.to, and the recommendations are always the same: put a number in the title, use all four tags, post on a Tuesday morning, add a cover image. I wanted to check those claims against the platform's own data, because dev.to has a public API and nobody seems to have pointed it at this question.

I got an answer, but not to the question I asked. Before you can ask what makes a post do well, you have to know what a normal post looks like, and it turns out almost nobody knows, including me, because the version of dev.to you read is not the one you publish into.

Two websites wearing the same logo

The feed at dev.to is curated. I pulled 2,500 posts from it and looked at the ones published in August, so everything had at least a fortnight to collect reactions. In that sample of 170 posts, the median post had 15 reactions, 70.6% of them had ten or more, and the mean was 28.8. Exactly zero of them had no reactions at all.

Then I sampled the site itself rather than the feed, by picking article IDs at random and fetching each one directly. That gives 460 posts published in August, chosen without any regard for whether anyone liked them. Here is that sample:

	curated feed	the actual site
posts sampled (August)	170	460
mean reactions	28.8	0.21
zero reactions	0.0%	91.7%
ten or more reactions	70.6%	0.4%
zero comments	—	95.7%

Ninety-two per cent of posts, plus or minus about two and a half points at this sample size, get no reaction whatsoever. Not a low number of reactions. None. And 95.7% of them get no comments either.

These are not bad posts. The zero-reaction pile, read at random, looks like this:

0 reactions  Four JavaScript problems I hit writing a chess engine
0 reactions  Delta E is a distance, not a score
0 reactions  How Farm.js compiles React components into direct DOM updates
0 reactions  Why Zod Crashed My Node.js Server

I would read all four of those. They went out into total silence, and the feed I actually see has been quietly filtering them out of my view for as long as I have been posting here.

Getting a fair sample is the hard part

My first attempt was the obvious one. dev.to's API lets you list articles by tag, so I paged through 39 popular tags and collected 82,749 posts. Then, before trusting any of it, I ran a check: the curated feed's August posts are popular by construction, so my big corpus ought to contain nearly all of them. It contained 49%.

So I took one missing post, a webdev article with 60 reactions published on 31 August, and walked twelve pages of the webdev tag listing looking for it. It is not there. What is there is stranger:

page 10: newest=2026-08-31  oldest=2026-08-29
page 11: newest=2026-09-13  oldest=2026-08-27
page 12: newest=2026-08-27  oldest=2026-08-18

Page 11 contains posts newer than anything on page 10. The pages overlap, they run backwards and forwards through time, and posts go missing between them. The listing is not an ordered, complete view of a tag, so you cannot page it to build a corpus, which is precisely what I had just spent forty minutes doing.

Worse, the omissions are not random. Comparing my random-ID sample against the tag corpus, the posts the listing gave me averaged 0.40 reactions while the ones it skipped averaged 0.15, and 19.0% of the included posts had at least one reaction against 5.1% of the excluded. Sampling through the listing quietly hands you a rosier site than the real one. My first draft of this post, written off that corpus, said 85% of posts get nothing. The true figure is 92%, and the error was entirely my instrument.

I also checked that a fortnight really is long enough for a post to finish accumulating. Splitting the August posts by age at the time I fetched them, the share getting any reaction was 13.6% at 14-20 days old and 15.0% at 42-48 days, with no trend in between, so nothing meaningful arrives after the first couple of weeks.

What actually correlates with anything

For the finer comparisons I used the large tag corpus, 45,400 August posts, because the random sample is too small to slice. It leans optimistic in absolute terms, for the reason above, but the relative differences between groups are what matter here.

The advice is not all wrong. A cover image goes with 23.7% of posts getting at least one reaction against 11.2% without. Writing in the first person, a title with "I" or "my" in it, goes with 24.7% against 13.9%. Length helps up to a point, with posts of three to seventeen minutes' reading time doing better than posts under three minutes. Four tags beat one, by 16.0% against 3.5%.

Two pieces of standard advice are simply backwards. Titles beginning with "How" did worse than titles that do not, 12.2% against 15.1%. Listicles, the numbered-list titles, got at least one reaction less often than other posts, 9.5% against 15.0%, though their mean was higher, 0.80 against 0.50, which is the signature of a format that mostly flops and occasionally spikes.

The tag you choose matters more than anything you do to the title. Among tags with at least 300 posts:

tag	posts	any reaction
css	560	38.8%
showdev	960	38.2%
frontend	541	33.6%
typescript	1,556	30.1%
crypto	926	1.1%
jobs	463	0.9%
defi	925	0.1%
tech	800	0.0%

Eight hundred posts tagged tech in one month, and not one of them got a single reaction.

And posting more is associated with doing worse per post, not better. Authors publishing one to five posts in August had 23.3% of them get a reaction; authors publishing 21 to 25 had 7.5%. Those high-volume authors are not a curiosity either, they produced 18,386 of the 45,400 posts in the corpus.

The part that undoes most of the above

All of those comparisons share a flaw. If experienced writers with existing followers are also the people who add cover images and write in the first person, then I am measuring the audience, not the technique.

So I ran it again inside each author. For every author with at least four posts in the month, I compared their own posts with a given feature against their own posts without it, then looked at the distribution of those paired differences. If a first-person title genuinely helps, an author's first-person posts should beat their other posts.

They do not. Across 254 authors, the median paired difference for first-person titles was −0.08 reactions, with a 95% confidence interval of [−0.13, +0.00]. Listicles came out at −0.08, [−0.12, −0.03]. Question-mark titles, −0.08, [−0.14, −0.04]. Colons, 0.00. Every apparent effect from the previous section collapses to nothing, or to very slightly negative, once each writer is compared against themselves.

The honest reading is that the title patterns were never doing the work. They were a marker of which kind of account was posting, and when you hold the account fixed they stop predicting anything. What survives the within-author test is not the phrasing, it is which tag you land in and whether anyone was already reading you.

There was one result I could not explain away, and the sample is small enough that I would not lean on it: the 273 posts carrying dev.to's new AI-assisted disclosure label averaged 4.06 reactions with 11.0% reaching ten, against 0.48 and 0.7% for everything else. That is either a real effect or, more likely, evidence that the people who bother setting a brand-new metadata field are unusually engaged with the platform to begin with.

Where I sit, since it would be cowardly not to say

My last twenty posts run from 5 to 16 reactions, median 8. Against the random sample, a post with 8 reactions beats 99.6% of dev.to. Sixteen beats everything in a 460-post draw.

That sounded like good news for about ten seconds. What it actually means is that the bar is on the floor: writing something that a handful of people react to puts you in the top half of one per cent, and the median experience of publishing here is being read by nobody at all.

What I got wrong

Twice, in the same afternoon, and both were instrumentation rather than analysis.

The first fetch piped its progress through tail, which buffers, so I watched an empty log for twenty minutes and assumed it had hung. It had not; it had collected 20,596 posts and was holding them in memory to write at the end. I killed it and lost all of them.

The second is the one that would have ruined the post. I had a complete-looking corpus of 82,749 posts and a headline number, and the only reason I did not publish 85% is that I made myself check the corpus against a list of posts I knew should be in it. If I had skipped that check, everything above would read the same and be wrong by seven percentage points, in the flattering direction. The check took four minutes.

What to take from it

If you write here, the relevant comparison is not the feed. The feed is the top fraction of a per cent, and measuring yourself against it is like judging your running against the people on television.

Pick your tag deliberately, because that is the one input with a large and consistent association. Put a cover image on it. Then stop optimising the title, because within a given author it does not appear to matter, and spend the time on the thing that does: being worth following, so that the next post starts with an audience rather than hoping to find one.

And if you are going to publish numbers about a platform, check your sample against something you already know the answer to. Mine was wrong in the direction that made the story nicer, which is the direction these things usually fail in.

Only 10 of the top 1000 sites hide the hostname you are visiting. I built the tool that counts.

Remdore — Tue, 15 Sep 2026 07:01:00 +0000

A few days ago I wrote about how the name of the site you visit over HTTPS travels across the network in plain text, sitting in the TLS ClientHello where anything on the path can read it. The reaction I did not expect was how many people wanted to know the same thing about their own sites, and had no quick way to find out. So I built the quick way, a small web app that answers one question: does this domain's hostname leak, and could it not?

You type a domain, it reads that domain's public DNS, and it tells you two things. Whether the site publishes an Encrypted ClientHello config, which is the only thing that can hide the hostname, and if it does, what cover name a watcher would see instead of the real one. It performs no handshake and needs no key, because the whole point of the original piece was that none of this is secret in the first place.

What it actually does

The interesting information lives entirely in DNS, which surprised me when I first worked it out. A site that supports ECH publishes its public key in a DNS record of type HTTPS, the same record that carries things like the ALPN list and IP hints. So the check is: fetch that record over DoH, look for an ech= parameter, and if it is there, decode it far enough to read the version, the key-exchange algorithm and the public_name. That is the cover hostname, the decoy that appears on the wire in place of wherever you were really going.

When a domain does publish ECH, the tool shows you the lot:

The whole check is one short function. It reads public DNS, sends nothing to the site you are asking about, and keeps no record of what you type, because there is nothing it would need them for. That also makes it cheap and safe to run, which matters for the next part.

So I pointed it at the top thousand sites

A single-domain checker answers your question about your site. The more interesting question is how common ECH actually is, so I fed the tool a list instead of a box. The command-line side of it takes a file of domains and reports how many publish ECH, and I ran it over the top 1000 domains by traffic.

Ten of them publish ECH. Ten, out of a thousand. That is one per cent, and 992 of the rest leak their hostname exactly the way the first post described, with the remainder failing to resolve.

The ten are the part worth staring at:

tinyurl.com          cover=cloudflare-ech.com
cedexis.net          cover=cloudflare-ech.com
sagepub.com          cover=cloudflare-ech.com
sxyprn.com           cover=cloudflare-ech.com
gizmodo.com          cover=cloudflare-ech.com
animeflv.net         cover=cloudflare-ech.com
remove.bg            cover=cloudflare-ech.com
arnebrachhold.de     cover=cloudflare-ech.com
yayoye-spiele.com    cover=cloudflare-ech.com

Every single one hides behind the same cover name, cloudflare-ech.com. Not one of them is Google or Amazon or Microsoft or a bank. They are a scattering of ordinary sites that happen to sit on Cloudflare, in zones where Cloudflare has switched the feature on. ECH in the top 1000 is not really a story about those sites choosing privacy; it is a story about one provider flipping a setting for some of its customers and not others. Even cloudflare.com itself, the apex, does not publish it, though the test host crypto.cloudflare.com does.

I pointed the tool at my own sites too, because it felt dishonest not to. extratime.world, a football quiz site I run, leaks its hostname just like the other 992. I have not turned ECH on either, and having now built the thing that measures it, I understand exactly why almost nobody has: it is off by default, it needs your DNS provider and your CDN to cooperate, and it only helps at all if the visitor's DNS is encrypted too. It is not a checkbox, it is a small alignment of four different parties, and the survey is what that looks like from the outside.

The number that is not the number

The tempting headline is "one per cent of the web supports ECH", and I want to be careful, because that is not quite what I measured. I measured the top 1000 by a particular traffic ranking, on one day, resolved through one DoH provider. A different list, or the same list next month, would give a different count, and the long tail of small Cloudflare-hosted sites almost certainly has a higher rate than the giants at the top. So the honest claim is narrower and more interesting than the headline: among the most-visited sites, the ones whose traffic a network observer would most want to profile, ECH is essentially absent, and where it exists it is one company's doing.

Where it ran, and where it is now

I hosted the live copy on a 1 vCPU DigitalOcean droplet in Frankfurt while I took the screenshots above, which is the honest way to check that a thing actually serves traffic rather than just runs on my laptop. I have since taken that droplet down, so there is no public instance to link you to and nothing accruing a bill.

That is deliberate, and it is the better shape for this kind of tool anyway. The whole thing is open source, it is one small container, and running your own copy is two commands:

git clone https://github.com/oceanforge/sni-leak-check
cd sni-leak-check && docker compose up --build

Then it is on localhost:8000 and it is yours. A tool that inspects your infrastructure is better run by you than pointed at someone else's server, and this one is small enough that there is no reason not to.

What I got wrong on the way

The first time I started the container to test it, the page that came back was not my app. It was the admin API of a completely different project of mine that happens to run on the same machine, because I had mapped my new container to port 8000 and something was already sitting there. The health check even passed, because there was a healthy service on that port, just not mine. For a confused minute I thought my app was serving someone else's HTML.

It is the same lesson that keeps recurring in these posts, wearing yet another costume: a green check is not proof that the thing you think you are testing is the thing that answered. The fix was to move to a port nothing else was using, and to actually read the page title rather than trust the 200. I now check the title.

Contributing

The app does the one job well and there is a lot of obvious room to make it better, so I have left a set of good first issues on the repository: showing the raw ECH bytes with each field labelled, a bulk-check table, a shareable permalink, actively capturing a real ClientHello to show the leak rather than inferring it. They are small and self-contained on purpose. If you have been looking for something to contribute to during Hacktoberfest, this is an open invitation, and the issues are labelled to match.

The takeaway

The original post argued that HTTPS keeps two different promises and only one of them, the privacy of your traffic, has been kept for most of the web's life. This tool is the argument turned into a thing you can run against any domain, including your own. Check a few of the sites you use every day. Almost all of them will tell you the same thing, that the padlock protects the conversation and leaves the guest list in plain sight, and now you can see it for yourself in about a second.

The hostname you visit over HTTPS is sent in plain text

Remdore — Tue, 15 Sep 2026 05:50:49 +0000

I had assumed, without ever checking, that the padlock meant nobody on the network could tell which site I was visiting. The connection is encrypted end to end, so surely the destination is part of what gets hidden. It is not. The name of the site you are asking for travels across the network in plain readable text, at the very start of every HTTPS connection, and I wanted to see it with my own eyes rather than take anyone's word for it.

So I captured the first thing my machine sends when it opens an HTTPS connection, the TLS ClientHello, and went looking for the hostname in the bytes. It was not hard to find.

The hostname, sitting in the packet

Here is the ClientHello openssl sent when I asked it to connect with the server name secret.wikipedia.example. This is the raw handshake, dumped as hex before any key exchange has happened:

01 00 00 d7 03 03 dd 4f 36 06 ...
... 00 00 1d 00 1b 00 00 18 73 65 63 72 65 74 2e 77
69 6b 69 70 65 64 69 61 2e 65 78 61 6d 70 6c 65 ...

Those bytes from 73 onward are ASCII. 73 65 63 72 65 74 is "secret", and it runs straight through to "secret.wikipedia.example". The hostname is not hashed, not encrypted, not obscured. It is the literal string, sitting in the clear inside a structure called the Server Name Indication extension.

The handful of bytes in front of the name are just the wrapping, and once you have read them once the whole thing stops being mysterious. The pair 00 00 is announcing an extension of type zero, which is the number TLS assigns to server_name. After that comes a length, 00 1d, twenty-nine bytes, then another length for the list inside it, then a single 00 marking the entry as a host_name rather than anything else, then the length of the name itself, 00 18, which is twenty-four. Count twenty-four bytes on and you have run exactly to the end of "secret.wikipedia.example". I wrote a few lines to do that walk so I would not be counting nibbles by hand, and it landed where the hex said it would:

server_name extension found at offset 104
host_name length 24 -> 'secret.wikipedia.example'

Notice what did not happen there. Nothing was decrypted, and nothing could have been, because the key exchange is still several messages away when this is sent. The name goes first, and it goes in the open, and that is the protocol working as intended rather than failing.

Why it is sent in the clear

This is not a bug or an oversight, which is the part that took me a moment to accept. The server needs to know which certificate to present, and a single IP address today serves hundreds or thousands of different sites behind one TLS terminator. The server has to be told which site you want before it can hand you the right certificate and set up encryption for it. The SNI is how it is told. The chicken-and-egg problem, that you cannot encrypt the request for a site with that site's key before you have asked for the site, is real, and for twenty-odd years the answer has simply been to send the name unprotected.

A passive listener needs no key and no handshake

To make sure I was not being fooled by openssl showing me its own internal state, I wrote a listener that does nothing but read. It accepts a connection, reads the first packet, and pulls the hostname out. It never completes the handshake, never has a private key, never decrypts anything. It is about fifteen lines. Pointed a client at it asking for login.mybank.example.com:

read 1566 bytes, first byte 0x16 (0x16 = TLS handshake)
hostname the client is asking for: 'login.mybank.example.com'
bytes decrypted to get it: 0

That is the entire capability a corporate proxy, a national firewall, or your ISP needs to know every site you visit: read the first packet, skip to offset-something, read the length-prefixed string. The connection contents stay perfectly encrypted and completely beside the point. This is exactly how filtering by domain works without anyone breaking the encryption, and it is why "but it's HTTPS" is not the privacy argument people think it is. They are not reading your traffic. They are reading the label on the envelope, and you wrote it in block capitals.

TLS 1.3 does not fix it

I genuinely expected TLS 1.3 to have closed this, because 1.3 encrypts far more of the handshake than 1.2 did, the certificate included. So I ran the same listener against a 1.3-only client asking for tls13.secret.example:

read 1489 bytes, first byte 0x16
hostname the client is asking for: 'tls13.secret.example'
bytes decrypted to get it: 0

Still there, still readable, still zero decryption. TLS 1.3 encrypts the server's certificate so a watcher cannot see which cert came back, but the ClientHello, the client's opening message, is by definition sent before any keys exist, and the SNI lives in the ClientHello. Moving it was left to a separate mechanism.

The actual fix, and where it stands

That separate mechanism is Encrypted Client Hello, ECH, and it works by solving the chicken-and-egg problem with DNS. The server publishes a public key ahead of time, in a DNS record, and the client fetches it before connecting and uses it to encrypt the real ClientHello, hostname and all, inside an outer one that carries only a generic cover name.

This is not theoretical, there are real ECH configs published right now. I pulled the DNS HTTPS record for crypto.cloudflare.com and decoded the ech= blob in it:

ECHConfig version : 0xfe0d  (the ECH standard)
config_id         : 192
KEM id            : 0x0020  (X25519)
public key        : 32 bytes, 06dbbb64d9e36721...
public_name (seen on the wire instead): 'cloudflare-ech.com'

That is the whole idea in one record. The 32-byte X25519 key is what the client encrypts the true hostname with, and cloudflare-ech.com is the public_name, the decoy that a passive listener would see in the SNI instead of wherever you were actually going. Everyone connecting to an ECH-protected Cloudflare site looks, from the network, like they are visiting the same single cover name.

Two things stopped me short of showing a working encrypted handshake, and both are the honest state of ECH rather than a limitation of my afternoon. The openssl I have, 3.5, has no ECH support in its s_client at all, so I could decode the published config but could not make a client use it. And adoption is patchy even at the vendor that invented it: the same DNS lookup for cloudflare.com itself, the apex, returned an HTTPS record with no ech= field. The test host has it, the flagship domain does not.

There is also a deeper catch worth naming. ECH moves the secret out of the TLS handshake, but it moves it into DNS, because the client has to look up that public key by name first. If that DNS query goes out in plaintext, which is still the default on most machines, the watcher just reads the hostname from the DNS request instead and nothing has been gained. ECH only buys you privacy when your DNS is also encrypted, so it is really one piece of a set, not a switch you flip.

What I got wrong on the way

My first plan for proving the hostname was on the wire, and not just inside openssl, was to capture the packet with tcpdump and grep the raw bytes for it. That failed immediately, because tcpdump needs privileges to put the interface in promiscuous mode and this machine had no passwordless way to give it them.

The fix turned out to be better than the plan. Instead of sniffing my own traffic from above, I wrote the fifteen-line listener and pointed the client at it, so the "attacker" was just an ordinary TCP server reading its input. That is not a weaker demonstration than a packet capture, it is a truer one, because a real SNI-based filter is not doing exotic packet capture either. It is a box in the path reading the first bytes of the stream, which is precisely what my little server was. The privilege I could not get turned out to be privilege the demonstration never needed.

See it yourself

There is nothing to install for this. If you have openssl you can watch your own machine announce its destination, right now:

printf 'Q\n' | openssl s_client -connect example.com:443 \
  -servername anything.you.like -msg 2>&1 | sed -n '/ClientHello/,/</p'

Look for the 00 00 extension in the hex, step past the length bytes the same way I did above, and whatever name you passed will be sitting there in ASCII. And if you want to know whether a given site has bothered to publish an ECH config yet, the answer is in its DNS HTTPS record:

curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=crypto.cloudflare.com&type=HTTPS' \
  | grep -o 'ech=[A-Za-z0-9+/=]*'

An ech= blob coming back means that host is able to hide its name from the network. Try it on a few of the sites you actually use, and most of them will hand you back nothing at all.

What to take away

The encryption in HTTPS is real and it is doing its job: the contents of your connection are private. But privacy of contents and privacy of destination are two different promises, and only the first one has been kept for most of the web's history. The name of every site you visit is still, today, sent in a form any box on the path can read without doing any work. ECH is the fix, it exists, and the config is sitting in DNS waiting, but it needs encrypted DNS beside it and it is not yet switched on for most of the places you go. Until it is, assume the network knows the guest list even though it cannot hear the conversation.

Follow-up: I turned this into a tool you can run against any domain, and surveyed the top 1000 sites with it. Only 10 hide the hostname.

A container is just a process with a few private views. I built one in thirty lines.

Remdore — Mon, 14 Sep 2026 08:50:00 +0000

Here is a thing that bothered me for years without my quite admitting it: I did not know what a container actually was. I knew how to run one. I could not have told you what the kernel does when you start one, because the honest answer is that there is no such kernel object to point at. A container is just a normal process that has been handed a few private versions of things it would otherwise share, and I only really believed that once I built one myself, by hand, in about thirty lines and with Docker nowhere in sight. The part that surprised me was how little there was to it.

The process I ended up with thought it was PID 1 on a machine called container. It could not see a single one of my other processes. It had its own network, empty, and its own root filesystem, and if it reached for more than 20MB of memory the kernel killed it. That is a container, all of it. And the thirty lines are not doing anything clever, they are just asking the kernel, one request at a time, for each of those private views.

Namespaces are the whole trick

The Linux feature underneath all of this is the namespace. A namespace is a private copy of one kind of global resource. There are several: PID, mount, network, UTS (the hostname), IPC, user. A normal process shares all of them with everything else on the machine. unshare is the command that says: give me a fresh one.

The entire isolation half of a container is one command:

unshare --user --map-root-user --mount --uts --ipc --pid --fork --net \
  python3 enter.py rootfs

Each flag is buying one kind of isolation and no more. --uts hands me a private hostname, which is why I can rename the machine to container and nothing on the host notices. --pid --fork gives a private process table, and because of the fork the child, not the unshare itself, is the one that becomes PID 1. --net drops me into an empty network stack: one loopback interface, down, and nothing else at all. --mount gives a private set of mounts, which is what lets me swap the filesystem later. The odd one out is --user --map-root-user, and it is the flag doing the quiet heavy lifting, because it makes a user namespace and maps my normal login to root inside it. That is what lets the process mount things and call pivot_root, while out on the host I am still nobody special.

Run that and look around inside, and the isolation is already total:

hostname: container
pid: 1
--- processes (own PID ns) ---
PID   USER     TIME  COMMAND
    1 root      0:00 /bin/sh
    4 root      0:00 ps -ef
--- net ---
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000

Two processes visible where the host has eighty-seven. One network interface, down. A hostname I chose. I did not install anything and I was never root on the host. That output is the entire point of the post: isolation is not something you build, it is something you ask the kernel for, one namespace at a time.

Swapping the root filesystem is the fiddly part

The one genuinely tricky step is giving the process a different root directory, because this is where the difference between a toy and a real container lives.

The obvious move is chroot, and it does work, but it is also the bit people warn you off, because a sufficiently privileged process can climb back out of a chroot. The old root is still mounted underneath, only hidden, and there are well-worn tricks for reaching it again. So the real runtimes reach for pivot_root instead. It swaps the root mount out and then lets you unmount the old one completely, and once the old root is unmounted there is simply nowhere left to escape to.

pivot_root is a syscall with no friendly wrapper on my machine, so enter.py calls it directly through libc. The sequence has three requirements that each cost me a run to discover. The new root has to be a mount point, so I bind-mount the rootfs onto itself. The parent mount has to be private, or pivot_root refuses with EINVAL, so I remount / as private first. And after the pivot the old root hangs under a directory I have to detach:

mount(None, "/", None, MS_REC | MS_PRIVATE)
mount(rootfs, rootfs, None, MS_BIND | MS_REC)
os.chdir(rootfs)
sc("pivot_root", b".", b"oldroot")
os.chroot("/")
sc("umount2", b"/oldroot", MNT_DETACH)   # the host filesystem is now gone

After that last line the container cannot see the host filesystem at all. Not hidden, gone. ls /oldroot returns "No such file or directory", and the only mounts left are the new root and a fresh /proc. That is the line that turns a chroot into a container.

The mistake that made ps lie

My first version used chroot and mounted /proc the easy way, with unshare --mount-proc. Inside, ps showed no processes at all, and for a happy minute I thought I had built the most isolated container in history.

I had not. ps reads /proc, and --mount-proc mounts it over /proc in the mount namespace before the chroot happens. After the chroot, the process is looking at rootfs/proc, which is a different, empty directory. ps was not seeing an empty process table, it was seeing an empty folder. The isolation looked perfect because the instrument was unplugged.

The fix is to mount /proc after the root swap, from inside the new root, so it is the container's /proc that reflects the container's PID namespace. Once I did that, ps showed exactly two processes, which is the real answer, and a much better one than zero, because zero was a lie and two is proof the PID namespace works.

Namespaces isolate. Cgroups limit.

Isolation is only half of what people mean by "container". The other half is the ceiling, the guarantee that the thing cannot run off with all the memory on the box, and namespaces do exactly nothing about that. That is a different kernel feature entirely, control groups, and once I had built the two halves separately the division stuck in my head: namespaces govern what a process can see, cgroups govern what it can take.

A cgroup v2 memory limit really is just a number in a file. You put the process into the cgroup, write a byte count into memory.max, and the kernel does the rest. To watch it actually bite I compiled a tiny static binary, dropped it into the rootfs, and had it grab a megabyte at a time, touching each one so the pages were genuinely resident rather than just promised, then set the cap to 20MB and let it run:

cap is 20MB. starting the memory hog as a child...
allocated 5 MB
allocated 10 MB
allocated 15 MB
Killed
hog exited with code 137

That exit code, 137, is 128 plus 9, so signal 9, a SIGKILL, and the sender was the kernel's out-of-memory killer firing the moment the cgroup went over its cap. The container's own shell was fine, still sitting there as PID 1, because the killer took only the greedy child and handed its memory back. So there it is: a real memory ceiling, enforced by the kernel, on a process I had isolated by hand a few minutes earlier.

Getting there cost me two more mistakes. My first memory hog filled a tmpfs instead of allocating anonymous memory, and tmpfs pages are not freed when the writing process dies, so the OOM killer killed the hog, the memory stayed full, and it went round again and killed PID 1, and the whole container collapsed. Anonymous memory that frees on death is the honest way to demonstrate a limit. And I could not attach the process to a cgroup by hand at all at first: on this machine my shell lives in a cgroup called /init.scope that I do not own, outside the tree systemd delegates to my user, so every attempt to move it returned EIO. The way through is systemd-run --user --scope, which starts the process inside the delegated tree where I am allowed to set limits. Which is itself the lesson: namespaces are genuinely unprivileged, cgroups need either root or a slice someone delegated to you.

What it is not

A container shares the host kernel, and once you have built one by hand you can feel exactly how thin the separation is. Inside my container, uname -r prints 6.6.114.1-microsoft-standard-WSL2, the host's kernel, because it is the host's kernel; there is only one, and every container on a machine is running on it. cat /proc/uptime inside reported the same 258,345 seconds as the host, because I did not ask for a time namespace and so the container shares the host's clock and boot time.

This is the whole difference between a container and a virtual machine, and it is not a detail. A VM brings its own kernel and the isolation goes down to the hardware. A container is your process with some of its views swapped out, running on the same kernel as everything else, and a kernel bug is a shared fate. That is the trade: containers are cheap because they are barely anything, and they are barely anything because they are just your process wearing a few borrowed namespaces.

Run it yourself

You need a root filesystem to run in. The easy way to get one is to borrow it from an image you already have:

mkdir rootfs
docker export $(docker create alpine:3.20) | tar -C rootfs -xf -

Then the launcher is the one unshare line above plus the enter.py that does the pivot, twenty-seven lines of it, calling mount, pivot_root and sethostname through libc. Run it and check the three things that prove it worked: hostname is what you set, ps shows one or two processes and not the host's hundred, and ls /oldroot fails because the host filesystem is gone. If ps shows nothing, your /proc is in the wrong place, which is the same mistake I made.

What to take away

The next time Docker feels like magic, remember that the isolating part of it is six flags to unshare and a pivot_root, and the limiting part is a number in a file. Everything else Docker gives you, images, layers, networking, a registry, is real engineering built on top, but the container itself, the thing people imagine as a sealed box, is a process that asked the kernel for a few private views and got them. It is worth building one once, because after you have, "it's just a process" stops being a slogan and starts being something you have watched happen.

ESLint took 4.4s to lint Vue's core. oxlint took 0.24s. Then I turned on the type-aware rules.

Remdore — Sun, 13 Sep 2026 15:33:00 +0000

ESLint took 4.4 seconds to lint 445 TypeScript files. oxlint took 0.24. Turn on the type-aware rules that most projects actually want and ESLint climbs to 12 seconds, while oxlint, doing the type-aware pass too, finishes in under one. I went in expecting to write that oxlint was fast but limited. The limited part did not survive the test.

The setup

oxlint is the JavaScript and TypeScript linter from VoidZero, the company Evan You started after Vue and Vite, written in Rust and pitched as a much faster ESLint. I wanted a real number on a real codebase, not a toy, so I cloned the Vue core repository, 445 TypeScript files and about 150,000 lines, and linted the same tree four ways in a clean Node 20 container.

Two tools, two modes each. ESLint 9 with typescript-eslint, once with the plain recommended rules and once with the type-aware recommendedTypeChecked set that needs the TypeScript compiler behind it. Then oxlint, once with its defaults and once with --type-aware. Every number below is a wall-clock median of repeated runs on the same files.

Linter	Mode	Time
ESLint	recommended (syntactic)	4.4s
ESLint	recommendedTypeChecked (type-aware)	12s
oxlint	default (syntactic)	0.24s
oxlint	--type-aware	0.9s

The syntactic number is 18x, and that undersells it

On the plain recommended rules, oxlint linted the tree in 0.24 seconds against ESLint's 4.4, about eighteen times faster. But even that includes Node starting up and npx resolving the binary. oxlint prints its own engine time at the end of a run, and for this codebase it was 75 milliseconds. The actual linting of 150,000 lines took less than a tenth of a second; the rest was process startup.

That is the difference between a linter you run in a pre-commit hook and one you run in CI and hope finishes. At 75 milliseconds you can lint on every keystroke. At 4.4 seconds you lint when you remember to, or you wait for the CI job to tell you.

Type-aware is where I was wrong

Here is the part I got wrong, and it is the more interesting result.

The rules developers actually reach for are the type-aware ones. no-floating-promises, no-misused-promises, await-thenable: the checks that catch the bugs that matter, and every one of them needs the type checker to know what a value actually is. That is also why they are slow. ESLint's type-aware pass on this codebase took 12 seconds, nearly triple its syntactic run, because it has to build a TypeScript program before it can lint a line.

I had written, in my head, the tidy conclusion: oxlint is fast because it skips this, it does the cheap syntactic rules and leaves the expensive type-aware ones to ESLint. A fast first pass, not a replacement.

Then I checked, and oxlint has a --type-aware flag. It runs the type-aware rules through a companion called tsgolint, and on the same codebase it finished in 0.9 seconds. Type-aware. Under a second. That is roughly thirteen times faster than ESLint doing the same class of work, and it is still five times faster than ESLint's cheap syntactic run.

So the story is not "oxlint is fast because it does less". oxlint doing the expensive, type-aware analysis beat ESLint doing the cheap, syntactic one, by a factor of five.

The honest caveats, because there are several

This is where I have to be careful, because a lint-time comparison is easy to make unfair.

The rule sets are not identical. ESLint's recommended set and oxlint's default set overlap heavily but are not the same list, so I am measuring the time to lint the same files to a comparable standard, not the time to run byte-for-byte identical checks. oxlint reported 96 rules on the syntactic run and 111 with type-awareness on; typescript-eslint's sets are their own count. If you need one specific ESLint rule or a custom plugin that only exists for ESLint, oxlint may simply not have it, and no speed number rescues a missing rule.

The type-aware mode is newer and needs help. It is behind a flag, it pulls in the separate oxlint-tsgolint package, and oxlint itself calls the type-checking part experimental. When I first ran --type-aware without that package it failed cleanly and told me what to install, which is the right behaviour, but it is a step, not a default. The 0.9-second number is real; it is just not what you get from a bare oxlint invocation.

And the type checker underneath is a reimplementation, not tsc. tsgolint is a Go-based TypeScript type checker, and the whole reason it is fast is that it is not the official compiler. For linting that is a reasonable trade, but it is a different implementation of the type rules, and "different implementation" is exactly the phrase that should make you test it against your own code before trusting it in CI.

Where a football site feels this

A TypeScript frontend like ExtraTime runs the linter constantly: on save in the editor, on every commit through a pre-commit hook, on every pull request in CI. At ESLint's 4 to 12 seconds, the pre-commit hook is the thing everyone eventually adds --no-verify to get around, and the CI lint job is a minute of a runner's life on every push. At oxlint's sub-second, the hook stops being an interruption and the CI line stops being something you notice. It is the same code being checked; the difference is whether checking it is fast enough that nobody tries to skip it.

What I got wrong on the way

I nearly published the wrong limitation. My draft, before I ran the last test, said oxlint's ceiling was that it does not do type-aware linting, that it is a fast syntactic pass and you still need ESLint for the rules that need types. That is the received wisdom about fast linters and I repeated it without checking.

It took one --help to find the --type-aware flag and one more run to get 0.9 seconds out of it. The lesson is the one that keeps recurring: the tidy limitation that makes your narrative balanced is exactly the claim you have not tested. I had a clean story about a fast-but-limited tool, and the limit was not there. The real caveats, the ones above, are messier and less quotable, which is usually how you can tell they are the real ones.

Run it yourself

git clone --depth 1 https://github.com/vuejs/core
cd core && npm i -D eslint typescript-eslint typescript oxlint oxlint-tsgolint

# ESLint, syntactic then type-aware
time npx eslint "packages/**/*.ts"
time npx eslint -c eslint.typed.mjs "packages/**/*.ts"

# oxlint, syntactic then type-aware
time npx oxlint packages
time npx oxlint --type-aware packages

Watch the number oxlint prints at the end of its own run, the "finished in Nms" line. That is the linting; everything else your time shows is Node waking up.

What to do about it

Add oxlint as the fast gate and keep the slow one where it earns its place. Run oxlint in the pre-commit hook and the first CI step, because at sub-second cost there is no reason not to, and it will catch the great majority of issues before anything slower starts. Keep ESLint for the specific rules or plugins oxlint does not have yet, and run it once in CI rather than on every save. The point is not to pick a winner. It is that the fast tool is now fast enough to run everywhere, and the only question left is which handful of checks still justify the slow one.

A warm pip install took me 13 seconds. uv took 56 milliseconds.

Remdore — Sun, 13 Sep 2026 14:33:00 +0000

A warm pip install of a normal web backend took 13 seconds on my machine. The same install with uv took 56 milliseconds. Not 56 milliseconds faster. 56 milliseconds total, for 63 packages, including numpy and pandas and pillow.

I did not believe it either, so I ran it enough times to be sure the number was real and not a no-op. It is real, and the reason it is real is the interesting part.

What I measured

uv is the Python package installer from Astral, the company behind the Ruff linter, written in Rust and aimed squarely at pip. The pitch is "10 to 100 times faster". I wanted to know where in that range real life lands, so I built a requirements file that looks like something you would actually deploy: FastAPI, uvicorn, SQLAlchemy, Alembic, Pydantic, Celery, Redis, pandas, numpy, pillow, the password and JWT libraries, gunicorn, twenty top-level packages that resolve to sixty-odd.

Then I installed it four ways, three times each, in a clean Python 3.12 container: pip and uv, each from a cold cache into a fresh virtual environment, and each from a warm cache into a fresh virtual environment. Cold means the download cache was wiped first, the state a CI runner is in without caching. Warm means the cache was kept but the environment rebuilt, the state your laptop is in all day.

	pip	uv	uv is
Cold cache	~26s	~5.4s	~4.8x faster
Warm cache	~13.2s	~0.056s	~236x faster

Both resolved the same versions. I checked FastAPI, pandas, numpy, SQLAlchemy and Pydantic across the two environments and they matched exactly, so this is not uv winning by installing less.

The warm number is the one that matters, and it needs explaining

Cold, uv is about five times faster, and that is mostly parallel downloads: pip fetches packages fairly sequentially, uv saturates the connection. Useful, not shocking.

Warm is where it stops being a speedup and becomes a different category. 13 seconds down to 56 milliseconds is not pip with the slow parts removed. It is a fundamentally different operation.

pip, even with every wheel already in its cache, still copies each one into the new environment and runs the install machinery for it. Sixty-three packages, sixty-three unpack-and-copy operations, and that is your 13 seconds. It is doing real filesystem work proportional to the size of your dependencies, every single time you build an environment.

uv keeps a content-addressed global store, and by default it populates a new environment by hard-linking to the files already in that store rather than copying them. A hard link is a directory entry, not a copy of the data, so the cost is metadata and almost nothing else. That is why sixty-three packages land in 56 milliseconds: nothing is being copied, the files are already on disk and uv is just pointing at them.

Once you see it that way the number stops being surprising and starts being obvious. The question is not why uv is fast. It is why we accepted copying our entire dependency tree on every environment build for so long.

The same trick pays off on disk. I built three separate environments from the same requirements and measured what they actually cost. The cache held 225MB of package files. The three environments, added on top, cost about 12MB of real disk between them, because every file in them is a link back to the cache rather than a copy. pip would have written three full copies, roughly 675MB, to hold the same three environments. If you keep several environments around, uv is not just faster to build them, it stores them for a fraction of the space.

The catch, and it is a real one

The hard-link trick only works when the cache and the environment live on the same filesystem. Links cannot cross a mount point, so the moment they are on different disks uv falls back to copying, and copying is what made pip slow.

This is not academic, it is the normal shape of CI. The runner restores your cache onto one mount and builds the environment on another, and there goes the hard link. So I measured the fallback directly, forcing uv into copy mode:

uv warm install	time
same filesystem (hard link)	0.08s
forced to copy (cross-filesystem)	0.32s

Copy mode is four times slower than linking, exactly as you would expect. But look at the absolute number: 0.32 seconds, against pip's 13. Even stripped of its best trick, forced to copy every file the way pip does, uv is still about forty times faster, because it is also resolving and unpacking in parallel Rust instead of sequential Python. The hard links are the spectacular part; they are not the whole advantage.

Where a football site would feel it

If you run a Python backend, the place this lands is CI. A site like ExtraTime rebuilds its environment on every push: every pull request, every deploy, every test run installs the dependencies from scratch. On pip that is 13 to 26 seconds of a runner sitting there copying pandas around, on every job, forever. Swap the installer and that line in the log goes to sub-second warm, or five seconds cold on a runner with no cache. It is not a heroic optimisation, it is a one-line change to the install command, and it gives back time on work you run hundreds of times a week.

What I got wrong on the way, and a real aside

My very first cold pip run took 268 seconds. The next three took 26. I nearly wrote down 268 as the cold number, which would have made pip look five times worse than it is.

It was not pip. It was the first network contact from a fresh container: cold DNS, TLS setup, PyPI connections warming up. Every cold run after that, with the download cache still wiped each time, settled at around 26 seconds. The lesson is the ordinary one, that the first measurement is often measuring your environment rather than the thing you care about, so throw it away and watch whether the number stabilises. But there is a real point hiding in it too: a genuinely cold CI runner, first job of the day, can pay that 268-second tax once, and neither tool fully saves you from it because it is the network, not the installer.

Run it yourself

Install uv, then race it against pip on your own requirements:

pip install uv

# cold: wipe caches first
rm -rf ~/.cache/pip; python -m venv /tmp/v
time /tmp/v/bin/pip install -r requirements.txt

uv cache clean; uv venv /tmp/uvv
time VIRTUAL_ENV=/tmp/uvv uv pip install -r requirements.txt

# warm: run each again into a fresh venv without clearing the cache

The warm run is the one to watch. Delete the environment, keep the cache, and install again. pip will take about the same time it took before. uv will finish before you have read the command back.

What to do about it

For a new project, uv is a straight swap and the CI time is free money. For an existing one, the honest friction is that uv is another tool in the chain and your lockfile and workflow assume pip, so the win has to beat the cost of the change. The way to decide is to run the warm test above on your real requirements. If your dependency tree is small the 13 seconds might not be worth touching. If you install pandas and friends on every CI job, you are copying the same hundred megabytes thousands of times a month, and hard links exist precisely so you do not have to.

I misspelt 12 footballers' names. Postgres full-text found 4, Meilisearch found all 12.

Remdore — Sun, 13 Sep 2026 13:33:00 +0000

Type "belerin" into most search boxes and you get nothing. The player is Héctor Bellerín, you dropped an accent and a letter, and the database shrugs. I wanted to know how much that costs and whether the usual answer, "just use Postgres full-text", actually holds. So I loaded 1,760 real footballers into three engines and typed their names badly on purpose.

The short version: Postgres full-text search found the right player in 4 of 12 misspelt queries. Meilisearch found all 12. And the interesting one in the middle, Postgres trigram search, found 8, but two of its answers were the wrong player returned with total confidence.

The setup

The data is every Premier League player from five seasons of the Fantasy league export, deduplicated to 1,760 unique names. Real names, with the accents and hyphens that make this hard: Bellerín, Kolašinac, Papastathopoulos, Cédric, Pépé.

Three engines, same 1,760 rows:

Postgres full-text, a GIN index over to_tsvector('simple', full_name), queried with plainto_tsquery.
Postgres trigram, the pg_trgm extension with a GIN index, queried with word similarity.
Meilisearch, the search engine from the French startup of the same name, indexed with its defaults and nothing tuned.

Then twelve queries, each one a name typed the way a real person types it: an accent dropped, a letter doubled, a syllable transposed.

Full-text search does not tolerate a single mistake

Here is the result that should change what you reach for. Postgres full-text, on the twelve typo queries:

Typed	Found the player?
bellerin	no
mustaffi	no
martinez	no
cedric soares	no
pepe	no
kolasinac	yes
ceballos	yes
sokratis	yes

It got four, and every one it got was a name with no accent that the user happened to spell correctly. to_tsvector lowercases and splits on word boundaries, and that is the entire extent of its forgiveness. "martinez" does not match "Martínez" because the í is a different character and full-text does no folding. "mustaffi" does not match "Mustafi" because they are simply different tokens. There is no fuzziness in it at all. It is an exact-token index with a case fold, and a search box built on it silently fails the moment a user misremembers a spelling, which for foreign names is most of the time.

Trigram is better, and that is where it gets dangerous

Reach for pg_trgm next and things improve. It chops each name into three-letter runs and scores how many two strings share, so "mustaffi" and "Mustafi" overlap on almost all of theirs and match. I tuned the word-similarity threshold to 0.45, which is the sort of number you land on after a few tries, and it got 8 of the 12. Some of the accented ones too, which surprised me.

But look at what it did with the four it got wrong:

Typed	Trigram's top hit	The right answer
martinez	Gabriel Martinelli	Emiliano Martínez
hector belerin	Michael Hector	Héctor Bellerín
pepe	Stipe Perica	Nicolas Pépé

It did not return nothing. It returned Martinelli for Martínez, and it returned him as the single best match, ranked above the correct player, with no signal that it was guessing. "hector belerin" resolved to Michael Hector because the token "hector" scored cleanly and the mangled "belerin" was ignored. For a user this is worse than an empty result. An empty result tells you to try again. A confident wrong answer tells you the player you are thinking of does not exist, or quietly enters the wrong one.

This is the trap with trigram similarity as a search backend. It always has a best match, because similarity is a continuous score and something is always closest. Tighten the threshold to cut the wrong answers and you lose real matches too. It has no notion of "I am not sure", which is the one thing a search box most needs.

Meilisearch got all twelve, and was faster

Indexed with defaults, no configuration, Meilisearch returned the correct player as the top hit for all twelve queries, accents and doubled letters and transpositions alike. It folds diacritics, tolerates a bounded number of typos scaled to word length, and ranks exact and prefix matches above fuzzy ones so it does not do the Martinelli thing.

It was also the fastest of the three:

Engine	Correct (of 12)	Latency
Meilisearch	12	under 1ms server, ~1.2ms over HTTP
Postgres trigram	8	~3.2ms
Postgres full-text	4	~0.4ms, but mostly finding nothing

The full-text number is fast because returning zero rows is cheap. Meilisearch's sub-millisecond figure is its own reported processing time; the ~1.2ms is what I measured over the network including the HTTP round trip on the same box.

Where this actually matters

The whole premise of a football quiz site like ExtraTime is that you have a player in mind and the game has to know which one. Career Path, The Scout, the guessing games: they live or die on turning "that Arsenal right-back, Spanish, belerin something" into Héctor Bellerín. On full-text that query returns nothing and the game feels broken. On trigram it might return Michael Hector and the game feels wrong, which is worse. This is exactly the class of lookup where the difference between the three engines is the difference between a working feature and a frustrating one.

The honest cost

None of this makes "just use Postgres" bad advice. It makes it advice with a boundary.

Meilisearch is a second service. It is another process to run, another thing to monitor, and critically another copy of your data that has to be kept in sync with the source of truth. Your players live in Postgres; Meilisearch holds a denormalised index of them, and every insert and update and delete now has to reach two systems. That is real operational weight, and for a search box that only ever gets exact IDs or correctly spelled terms it is weight you do not need.

The line is roughly this: if search is a convenience, full-text in the database you already run is fine, and trigram covers a bit more as long as you are comfortable showing the occasional wrong match. If search is the product, if the user's whole task is finding the right thing from a fuzzy memory of it, a dedicated engine stops being over-engineering and starts being the thing that makes the feature work.

What I got wrong on the way

My first trigram run reported 0 out of 12, and I nearly wrote that Postgres trigram was useless. It was my bug. I had written the similarity operator as %% instead of %, doubling it out of habit, and Postgres rejected every query as an unknown operator. The benchmark counted each error as a miss and produced a clean, wrong, damning result: zero.

The tell was that it was too clean. Trigram scoring "mustaffi" against "Mustafi" is basic and I could see by hand it should match. When a whole column of results is a perfect zero, the instrument is usually broken, not the thing under test. I fixed the operator, tuned the threshold the way anyone deploying this would, and trigram came back with a fair 8 out of 12. Reporting the 0 would have been an easy, satisfying, completely false story.

Run it yourself

Meilisearch is one container and the index is one POST:

docker run -d -p 7700:7700 -e MEILI_MASTER_KEY=devkey getmeili/meilisearch:v1.12
curl -X POST localhost:7700/indexes/players/documents \
  -H 'Authorization: Bearer devkey' -H 'Content-Type: application/json' \
  --data-binary @players.json
curl 'localhost:7700/indexes/players/search' -H 'Authorization: Bearer devkey' \
  -H 'Content-Type: application/json' --data '{"q":"belerin"}'

For the Postgres side, the two indexes to compare are:

CREATE INDEX ON players USING gin (to_tsvector('simple', full_name));
CREATE INDEX ON players USING gin (full_name gin_trgm_ops);

Query the trigram one with SET pg_trgm.word_similarity_threshold = 0.45; and WHERE 'belerin' <% full_name ORDER BY word_similarity('belerin', full_name) DESC. Type real names badly and watch which engine finds the person you meant.

What to do about it

Try one typo before you decide. Take the search your users actually run, misspell it once the way a hurried person would, and see what each option returns. If full-text gives you nothing and you are fine with that because your users paste exact values, keep it. If trigram gives you a plausible wrong answer, notice that you would have shipped it. And if the entire point of the feature is finding a half-remembered thing, the dedicated engine is not the heavy option, it is the one that does the job.

My benchmark harness was wrong fourteen ways before it measured anything

Remdore — Sat, 12 Sep 2026 14:48:33 +0000

I built a harness to measure whether reverse proxies buffer Server-Sent
Events. The results are in the last post.
This post is about the harness, which was wrong in fourteen ways before it
produced a single number I would stand behind.

That is not a confession of sloppiness. It is the normal state of measurement
code, and the reason it stays wrong is structural: a benchmark harness is the
one piece of software whose output nobody can independently check. If your web
app returns the wrong price, a user complains. If your harness returns 1.02
instead of 1.00, it goes in a blog post and gets quoted back at you for two
years.

So here is every defect I found in mine, the lie each one would have told, and
the specific experiment that proved the fix worked. That last column is the
point of the whole post.

The worst one was invisible in the output

The suite runs several cells, each measuring six endpoints ten times. Two
guards protect it: one audits whether the emitter actually paced frames at the
interval it was told, the other requires a direct unproxied baseline to look
clean.

On an early run the pacing guard failed, so I ran it again. It failed again. It
passed on the sixth attempt, and I had a full table of clean-looking numbers.

Nothing in that table recorded that it took six attempts.

Re-running a measurement until the guard passes selects for moments when the
machine happened to be quiet. It is not a small bias, either. When I removed
the retry and took exactly one honest attempt, one cell out of six
certified. The other five were unmeasurable on that host. Six tries had
converted "this laptop cannot measure this" into a publishable table.

The falsification: run once, count what survives. Six of six became one of
six.

The fix was not just to stop retrying. It was to make the policy visible: the
suite now states in its own output that it makes exactly one attempt per cell
and never retries, and a cell that fails is labelled UNMEASURABLE with the
guard's own reason instead of quietly absent.

The guard had a tenth of the power it appeared to have

A reviewer found this one and it is my favourite defect of the set.

The runner audited the emitter's pacing after each cell. The emitter stores its
send log keyed by a request id. The runner used one id for all ten runs,
and the emitter overwrote the log on every stream.

So the guard audited the last run. The other nine went into the median
unaudited. It looked like ten runs of protection and it was one.

The falsification: give each run its own id and print the ids actually
audited. Three runs, three ids, three independent drift figures. Before the
fix there was one.

It blamed a proxy for its own stalls

The pacing guard recorded a timestamp after each write to the client. Writes
block. So when a buffering proxy applied backpressure, the emitter's write
blocked, and the guard recorded that as emitter drift — which fails the
guard and voids the cell.

Read that again in terms of what it does to a result. The cells most likely to
contain the finding are the cells most likely to be thrown away for
instrument error.

The fix: record the timestamp before the write, so the log measures when
the emitter woke up on schedule rather than when the downstream deigned to
accept bytes.

It perturbed the thing it was measuring

The emitter's Docker healthcheck spawned a fresh CPython interpreter, inside
the container, once per second, forever. Next to a loop whose entire job is
millisecond-accurate pacing.

It is obvious written down. It was invisible in a compose file.

The falsification: the drift spikes that voided five cells were 10.44 to
10.75ms against a 10.00ms tolerance. Cheap, infrequent probe instead, and the
same host measured 0.22ms.

The metric's core logic had no test at all

The headline number is frames per read: SSE frames received divided by the
number of recv() calls that delivered at least one. A frame arriving alone
gives 1.0. Forty-one frames in a single read gives 41.0.

Everything rests on incrementing the arrival counter once per read, not
once per frame. That distinction is the entire metric.

There was no test for it. Four tests covered the client, and all four used a
naturally-incremental stream where per-frame and per-read give the same answer.
A client with the increment in the wrong place passed all four.

The falsification: I wrote a relay that drains an entire upstream response
and then flushes it in one go, and asserted both sides of the contrast —
buffered at 21.00, direct at 1.00. Then I moved the increment inside the
per-frame loop and confirmed the test fails. It reports 1 arrival where 21 are
expected.

That test now pins the metric in git. The version of it I ran by hand, before
committing it, proved nothing to anyone but me.

It undercounted frames, silently

The SSE frame counter looked for blank-line terminators across reads, keeping
a tail of unconsumed bytes between calls. It counted with a non-overlapping
scan and trimmed the tail with a rightmost search. Those two can disagree
about which bytes a terminator occupied.

Given three or more consecutive newlines split across a read boundary, the tail
was trimmed past a newline the counter had not consumed, and frames were
lost. Not mangled, not errored — quietly absent from the count that every
published number derives from.

s = b"data: a\n\n" + b"\n\n" + b"data: b\n\n"   # event, blank-line keep-alive, event
s.count(b"\n\n")               -> 3
one read                       -> 3   correct
split at offset 10             -> 2   one event gone

A blank-line keep-alive next to an event boundary produces exactly this shape,
and real SSE endpoints send those.

The falsification: a 20,000-trial fuzz over an alphabet of only \n and
X, asserting the split total always equals the whole-string count. Zero
mismatches after the fix.

My own fuzz had passed this bug. I had built it from realistic SSE payloads,
which never generate three consecutive newlines. A fuzz alphabet has to
include the delimiter you are testing, not just plausible data.

The guard trusted the log it was auditing

The pacing audit sorted the emitter's send log by sequence number, then used
each entry's position as its expected time slot. It never checked the sequence
was complete.

Feed it seq [0,1,1,2] or [0,1,3,4] and it mapped entries to the wrong
planned timestamps and returned pass.

This is the function whose entire purpose is to not trust the instrument.

The falsification: both malformed logs now exit non-zero with "send log is
malformed", and a contiguous log still passes — so the check cannot be
satisfied by rejecting everything.

The self-test certified a server it had not started

selftest.sh launched the emitter in the background, waited for /healthz to
answer, then certified the host.

Three compounding mistakes. The launch was backgrounded, so a bind failure
exited a background job and set -e never saw it. The readiness probe was
curl /healthz, which any listener satisfies. And nothing ever checked the
process it launched was still alive.

I found it because a run printed Address already in use and then reached
== rig OK. A leftover emitter from an earlier run was squatting on the port.
The numbers happened to be valid. The script could not have known.

The falsification: a twelve-line decoy server that answers /healthz with
ok and does nothing else. It used to earn a rig-OK. It now gets refused with
"port already occupied".

A self-test that can certify against an unknown process is worse than no
self-test, because it produces confidence instead of an error.

Two smaller ones, same shape

The tests could not tell a median from a mean. Every gap sequence in the
metrics tests was uniform or all-zero, so median, mean and max were
indistinguishable. Swapping statistics.median for statistics.mean passed
all seven tests. Falsification: gaps of [10,10,10,200], where median is 10,
mean is 57.5 and max is 200, so each is separately falsifiable.

The leak guard was blind to the only secret in play. It grepped the tree
for the DigitalOcean API token prefix dop_v1_. The credential this code path
actually handles is a model access key, prefixed doo_v1_. Three letters, not
two. Falsification: planting a doo_v1_ string left the guard reporting "no
token material in the tree".

Then I built a mechanism that did not exist

This is the part I would most like to skip, which is how I know it belongs
here.

Measuring on macOS, I saw Caddy and Traefik take about 42ms to first token
while nginx took 2ms. The first request after startup was fast; every reused
one was slow. That is the signature of a delayed-ACK stall on a pooled
upstream connection. I had a mechanism, I could name which condition would
reproduce it, and I built a 2×2 to demonstrate it. One cell hit — warm pool,
Nagle enabled, 42.39ms — exactly where predicted and nowhere else.

Three things then dismantled it.

A reviewer pointed out my nginx config has no upstream{} block and no
keepalive, so nginx never pools upstream connections at all. It sat
permanently in the "fresh" regime that Caddy and Traefik only reach on their
first request. I had not been comparing proxies. I had been comparing pooled
against unpooled and reading the difference as a proxy property.

Then I ran the emitter alone — no proxy, no Docker, nothing in the path — for
40 runs. It produced 30 to 45ms spikes by itself. p90 of 10.89ms, max of
44.50ms.

Then I moved to Linux and the effect vanished across all nine cells and every
condition, pooled or fresh, Nagle on or off.

It was a Docker Desktop artifact. I had a mechanism, a prediction, and a
confirming observation, and the thing did not exist. The single hit in the
predicted cell was chance, and one observation is not a finding no matter how
well it fits the story you already have.

What actually made the difference

Not care. I was being careful the entire time.

Falsify every fix. For each defect above, the question was not "does it
pass now" but "does the test fail when I reintroduce the bug". Substitute the
mean and watch it fail. Move the increment per-frame and watch it fail. Plant
the token prefix and watch it fail. A fix you cannot falsify is a fix you are
taking on faith.

Isolate before attributing. Every wrong mechanism I built came from
measuring a composite and blaming one component. The emitter-alone run settled
in ten minutes what a 2×2 had failed to settle.

Make the exclusion rule declared, mechanical, and published. I do still
discard individual runs — 1.7% to 11.7% per cell, because across roughly 2,400
timed writes the chance of one scheduling hiccup approaches certainty. That is
legitimate where retry-until-green was not, and the difference is exactly four
properties: the criterion is stated in advance, it is mechanical, it is
measured on the instrument side independently of the result being tested, and
every exclusion is counted in the published table. Retrying until green fails
all four.

The check that convinced me: re-enabling Nagle on the emitter doubled the
exclusion rate from 5.0% to 10.0% and left every proxy number unchanged. The
exclusions were removing apparatus noise, not shaping the answer.

Write down the tolerance reasoning. My first guard used a flat millisecond
budget. At a 5ms emit interval a flat 5ms permits 100% error; a flat 10%
demands 0.2ms, which is below the scheduling granularity of a shared vCPU.
Neither is a tolerance, they are just numbers. It became max(2ms, 20% of interval) and the behaviour is now falsifiable in both directions: 4ms of
drift fails at a 5ms interval while 1.5ms passes, and 25ms fails at 50ms.

Publish the numbers that are not findings. Frames per read has a noise
floor around 1.02 — the direct, unproxied path measures 1.02. Quoting 1.02
for a proxy as if it differed from 1.00 is reading noise as signal, and I only
know that because something forced me to look at the baseline column.

The uncomfortable part

Half of these were found by review, not by me. The guard auditing one run in
ten, the timestamp after the blocking write, the nginx pooling confound that
killed my best mechanism — all three came from someone reading the whole diff
at once and asking what a number was allowed to prove.

My own pre-flight check on the plan had passed the task whose code could not
pass its own tests. I had verified that code and tests both existed and were
plausible, not that the code would actually pass the tests it shipped with.
Tracing assertions one at a time is the only version of that check worth
running.

The harness is fine now. Nine cells certify, the guards fail closed, and the
metric is pinned in git by a test that fails when you break it. But it took
three Critical and fourteen Important defects to get there, and the interesting
ones were never the coding errors. They were the measurement errors: a guard
with a tenth of its advertised power, a timestamp on the wrong side of a
blocking write, a healthcheck perturbing the thing it checked, and a retry loop
doing to my own numbers precisely what the harness existed to prevent.

If you have a benchmark you have never tried to break, you do not have a
benchmark. You have a number.

nginx streams your tokens fine. HAProxy holds them for 206ms.

Remdore — Sat, 12 Sep 2026 14:40:55 +0000

The advice is everywhere: if you are streaming Server-Sent Events through
nginx, turn off proxy_buffering or your tokens will arrive in one lump.

I built a rig to measure it. nginx does not buffer a token stream. The proxy
that does is HAProxy, on a stock config, and it holds your first token for
206 milliseconds. The header people reach for to fix SSE buffering has no
effect on it, because that header is an nginx convention and nginx was never
the problem.

Four proxies, pinned to exact patch versions, all in front of the same
emitter, measured on a Linux droplet:

proxy	time to first token	gap between frames	frames per read
direct (no proxy)	2ms	50.0ms	1.02
nginx 1.31.5	3ms	50.0ms	1.02
nginx 1.31.5, `proxy_buffering off`	2ms	50.0ms	1.02
Caddy 2.11.4	2ms	50.0ms	1.02
Traefik v3.7.13	2ms	50.0ms	1.02
HAProxy 3.4.4	206ms	0.0ms	5.12

The emitter sends a frame every 50ms. Through nginx, Caddy and Traefik the
frames arrive 50ms apart, one per read, indistinguishable from no proxy at
all. Through HAProxy they arrive in bursts of five with zero gap between
them, 206ms late.

proxy_buffering is on by default in that nginx row. I checked with
nginx -T inside the running container rather than trusting the config file.
Turning it off changes nothing.

The metric

Everything here rests on one number: frames per read. Take the number of
SSE frames the client received, divide by the number of recv() calls that
delivered at least one. 1.0 means every frame arrived on its own. 41.0 means
the entire stream landed in a single read.

It is a ratio, so it needs no baseline to interpret and it survives a noisy
host in a way that absolute latency does not.

Before trusting it I pointed it at a relay that deliberately drains an entire
upstream response and then flushes it in one go. Direct: 1.00. Through the
relay: 21.00. The instrument detects coalescing when coalescing is there.

Two numbers in this post are not findings, and I want that on the record
before the tables start:

Frames per read has a noise floor around 1.02. The direct, unproxied path measures 1.02. One merged read in 41 is the client and the kernel, not a proxy. Anyone reporting 1.02 as distinct from 1.00 is reading noise.
This host produces isolated 30-45ms scheduling spikes with no proxy and no Docker in the path at all. Emitter alone, 40 runs: p90 of 10.89ms, max of 44.50ms. So no ~40ms first-token stall can be pinned on a proxy from this rig. More on that below, because I nearly published one.

Token-sized frames are HAProxy's worst case

This is the part that makes it matter for anyone streaming from a model.

Same 50ms interval, same HAProxy, same stock config. The only change is
growing each frame from about 60 bytes to about 1.1KB:

frame size	time to first token	gap between frames	frames per read
~60 bytes	206ms	0.0ms	5.12
~1.1KB	53ms	41.0ms	1.46

Grow the payload and the coalescing largely goes away. First token four times
faster, frames arriving spaced near the emit interval instead of in bursts,
close to one frame per read.

Varying the emit rate points the same direction. Frames per read goes 13.67 at
a 5ms interval, 5.12 at 50ms, 2.05 at 200ms. Faster emission means more bytes
accumulating per unit time, and more bytes means earlier flushes.

So the trigger is substantially about filling a buffer, not a fixed timer. I
am not going to name a byte threshold: the values implied by different cells
do not reconcile cleanly, around 2.2KB from one and 1.05KB from another, and
pinning it down properly needs a dedicated sweep I have not run.

But the practical consequence is sharp. An LLM token stream is small frames
arriving steadily, which is exactly the shape that triggers this. Benchmark
the same proxy with realistic 1KB chunks and you will not see it. That is
probably why it is not better known.

It also means the frames-per-read figure is a property of this stream through
this proxy, not a property of HAProxy. Quoting "HAProxy delivers 5 events per
read" as a fact about HAProxy would be wrong; it delivers 13.67 per read at a
5ms interval and 2.05 at 200ms.

The header everyone reaches for does nothing

X-Accel-Buffering: no is the documented escape hatch for exactly this
problem. I sent it and measured again:

	time to first token	frames per read	max gap
HAProxy, no header	206ms	5.12	256.4ms
HAProxy, `X-Accel-Buffering: no`	214ms	5.12	256.2ms

Identical. The 206 against 214ms is run-to-run variation; the ratio and the
max gap match to three significant figures.

It is an nginx convention. nginx honours it, and nginx was not buffering.
HAProxy has never claimed to read it. So the standard fix for SSE buffering
is a header that the only buffering proxy in this set ignores.

When `proxy_buffering` does cost you something

I could not leave the nginx result as "the advice is unnecessary", because
every cell above uses a client that reads promptly and a stream of about
2.4KB total. nginx's default proxy_buffers is 8 × 4k or 8k, so 32 to 64KB.
The buffers were never close to full. The directive had nothing to do.

So I built the case where it does: about 328KB of payload, pushed through a
client that deliberately sleeps 200ms between reads.

endpoint	time to first token	frames per read
direct	2ms	3.73
nginx (`proxy_buffering` on)	53ms	3.73
nginx (`proxy_buffering off`)	3ms	3.57

There it is. Buffering on costs about 50ms of first-token latency, and it is
consistent: median 53ms, max 54ms across ten runs, never once fast.

Note what did not happen. Frames per read is 3.73 either way, the same as
direct. The coalescing in that column is the slow client and the kernel. The
directive cost latency; it did not turn the stream into one lump.

So the honest position on proxy_buffering is neither of the two things the
internet tells you:

Small frames, prompt client — the actual token-streaming case — it costs nothing measurable, across four separate cells.
Bulk payloads, slow client — it costs about 50ms of first-token latency.
In no cell did it cause the dramatic coalescing it is famous for.

The finding I killed

Early on, measuring on macOS with Docker Desktop, I saw Caddy and Traefik
consistently take about 42ms to first token while nginx took 2ms. The first
request after startup was fast and every reused one was slow, which is the
signature of a delayed-ACK stall on a pooled upstream connection. I had a
mechanism, I could predict which condition would reproduce it, and I built a
2×2 to demonstrate it.

Three things then went wrong with that story, in order.

A reviewer pointed out that my nginx config has no upstream{} block and no
keepalive, so nginx never pools upstream connections at all. It was
permanently in the "fresh connection" regime that Caddy and Traefik only hit
on their first request. I had not been comparing proxies. I had been comparing
pooled against unpooled.

Then I ran the emitter on its own, with no proxy and no Docker in the path,
and it produced 30 to 45ms spikes by itself. Whatever I had been measuring, I
could not attribute it to a proxy.

Then I moved to Linux and it vanished entirely. Across all nine cells and
every condition — pooled or fresh, Nagle on or off — Caddy and Traefik sit at
2 to 5ms. The fresh-conn cell restarts the whole stack five times so every
row is a genuine first request: Caddy 5ms, Traefik 4ms. No difference from
pooled.

It was a Docker Desktop measurement artifact. Reporting it as a proxy
behaviour would have been wrong, and I got within one experiment of doing it.

Does it hold against a real model

Every number above comes from a synthetic emitter, because exact pacing is
what makes the comparison causal. So I swapped the upstream for DigitalOcean's
serverless inference endpoint and measured through the same client, 15 calls,
all HTTP 200:

endpoint	frames per read	gap between frames	frames
HAProxy	5.43	0.0ms	169
nginx	1.04	8.6ms	185
nginx, `proxy_buffering off`	1.02	8.7ms	176

Synthetic said 5.12 against 1.02. Real tokens from gpt-oss-20b say 5.43
against 1.04.

This table cannot prove causation on its own — a model's time to first token
varies between calls, which is why the synthetic cells carry the argument. Its
only job is to show the effect is not an artifact of my emitter. It is not.

How the numbers were kept honest

Two guards gate everything, and both can void a result.

The emitter records its own send timestamps and exposes them, so after every
run the harness checks whether the emitter actually paced as instructed. If it
drifted, that run is discarded rather than blamed on a proxy. The direct
unproxied path also has to show a clean incremental stream, or no proxy row
from that host means anything.

The suite makes exactly one attempt per cell and never retries. That rule
exists because I broke it. An early run "succeeded" on the sixth attempt, and
when I stopped retrying, one honest attempt certified one cell out of six.
Re-running until the guards pass selects for quiet moments on the machine and
biases every number, invisibly.

Individual runs are discarded, at a rate of 1.7% to 11.7% depending on the
cell, because across roughly 2,400 timed writes per cell the chance of one
scheduling hiccup approaches certainty. That exclusion is only defensible
because it is declared in advance, mechanical, measured on the emitter side
independently of the proxy being tested, and every exclusion is counted in the
published table. Re-running until green fails all four of those tests.

The internal check that convinced me it was working: re-enabling Nagle on the
emitter doubled the exclusion rate from 5.0% to 10.0% and left every proxy
number unchanged. Exclusions were removing apparatus noise, not shaping the
result.

Numbers come from a 2-vCPU Ubuntu 24.04 droplet, not a laptop. Docker Desktop
on macOS routes container traffic through a VM network stack, which puts tens
of milliseconds of noise into a measurement whose signal is 50ms — and, as
above, invented a finding.

That is the short version. The harness itself was wrong in fourteen ways
before it produced a number I would stand behind: a guard that audited one run
in ten, a pacing timestamp on the wrong side of a blocking write, a healthcheck
perturbing the thing it was checking, and the retry loop above doing to my own
numbers precisely what the guards existed to prevent. Every one of those, and
the experiment that proved each fix actually bites, is written up separately in
My benchmark harness was wrong fourteen ways before it measured anything.

What I would still test

The exact flush trigger in HAProxy. A payload-size sweep at a fixed interval would settle whether it is a byte threshold, a timer, or both.
Whether option http-no-delay removes it, and what that costs.
Caddy and Traefik with upstream keepalive explicitly configured, since my configs leave pooling at vendor defaults and nginx's default is no pooling at all.

The harness is stdlib Python and Docker, no dependencies. selftest.sh
certifies the host before any measurement and refuses to run against a server
it did not start. run.sh writes the image digests and the full text of every
proxy config into the results file, so the configs in this post cannot drift
from the ones that produced these numbers.

If you are streaming tokens through HAProxy on a stock config, that is where
your first 206 milliseconds went.

Next.js 16's Turbopack is 2.5x faster, but the bundler is only 4% of your build

Remdore — Fri, 11 Sep 2026 11:05:00 +0000

Next.js 16 makes Turbopack the default bundler and puts a number on it: builds 2 to 5 times faster than webpack. I wanted to know whether that holds on something bigger than the starter template, and where the time actually goes, so I built a 60-route app and timed every phase of the build with both bundlers.

The headline is true. It is also the least interesting thing I found. The claim describes the bundler, and on a warm build the bundler is about 4% of your build time. The rest is spent in places Turbopack does not touch, and one of them is a six-second phase with no progress bar that nobody talks about.

The app

Not a hello-world. Sixty route pages, each pulling in shared components: a Recharts line chart, a Markdown renderer, a client-side toolbar with lucide-react icons, date-fns, and a zod schema. About 2,000 lines across pages and components, on Next.js 16.3.4 and React 19.2. It builds cleanly and identically on both bundlers, which is the point: same input, swap only --turbopack for --webpack.

Every number below is a median of three runs, on the same machine, back to back.

Cold builds: the claim holds, at the low end

A cold build is no .next directory at all, the state your CI runs in unless you cache between jobs.

	Turbopack	webpack	ratio
Compile step	2.9s	7.2s	2.5x
Total build (wall)	7.1s	18.8s	2.6x

So 2.5x on a cold compile. That is real, it is repeatable, and it sits at the bottom of the advertised 2-to-5x range rather than the top. If you were promised 5x on your CI and you do not cache anything between runs, you will feel closer to 2.5.

Warm builds: the cache is the real story

A warm build keeps .next/cache from a previous run and rebuilds the rest. This is what your laptop does all day, and what CI does if you cache the directory.

	Turbopack	webpack	ratio
Compile step	143ms	2.75s	19x
Total build (wall)	3.45s	14.0s	4.1x

The compile ratio jumps to nineteen. Turbopack's persistent cache takes the bundling step from three seconds to under a sixth of one. This is the number worth caring about, because it is the one you hit dozens of times a day, and it is well outside the range on the tin.

But look at the two rows. The compile got 19x faster and the total build only got 4x faster. That gap is the whole point of this post.

The bundler is 4% of your build

On the warm Turbopack build, the compile is 143 milliseconds out of a 3.45 second build. Here is where the rest goes:

Phase	Turbopack warm	Does the bundler own it?
Compile	143ms	yes
TypeScript check	950ms	no
Static generation (64 pages)	380ms	no
Page data, finalizing, process startup	~2s	no

Turbopack made the one part it controls almost free, and the build is still three and a half seconds because type checking, static generation and Next's own orchestration do not get faster. Make the bundler infinitely fast and this build still takes over three seconds.

This is the honest reading of "2-5x faster builds". The bundler is 2.5x to 19x faster depending on cache state, but the total build time you actually wait for improves less, and improves less the more pages you have to statically generate. On this app it was 4x warm. On an app with 600 pages instead of 60, the static generation phase grows and the bundler's share shrinks further.

The six seconds nobody mentions

The strangest result was in webpack's total, which did not add up from its phases. Compile 2.8s, TypeScript 1s, static generation 0.4s: that is about four seconds accounted for, out of a fourteen second build. I timestamped every line to find the missing ten.

Most of it is one phase: Collecting build traces, 6.6 seconds, with no progress bar and no mention in any benchmark I have seen.

[  4.1s] ✓ Compiled successfully in 2.7s
[  5.2s]   Finished TypeScript in 1069ms
[  6.8s] ✓ Generating static pages (64/64) in 429ms
[  7.3s]   Collecting build traces ...
[ 13.9s] Route (app)          <- 6.6 seconds later

Build traces are the file-dependency graph Next computes so a deployment knows which files each route needs. On webpack it cost 6.6 seconds on this app. Turbopack emits no such phase at all: I grepped its entire output and there is no "Collecting build traces" line. Whatever it does instead is folded in and invisible.

So half of webpack's warm build is a single phase that has nothing to do with compiling your code, and switching to Turbopack removes it. That is a bigger share of the win than the compile speedup, and no one advertises it.

Two more things worth knowing

The cache is bundler-specific. The first time I measured a warm Turbopack build, the compile came back at 3.1 seconds instead of 143 milliseconds. The cause was that the previous build had been webpack, and its cache does not carry over. Turbopack rebuilt from cold while believing it was warm. If you switch bundlers, or bounce between them in CI, your first build pays the full cold cost and there is no warning that it has.

The output is half the size. Turbopack's .next came out at 81 MB against webpack's 168 MB for the same app. I did not go chasing why, but if you cache the build output between CI runs, that is half the bytes to store and move.

What I got wrong on the way

I contaminated my own cache measurement and nearly reported the wrong number. My benchmark cleared everything in .next except the cache directory between runs, which is correct, but I ran Turbopack immediately after a webpack build and got a "warm" compile of 3.1 seconds. I almost wrote that Turbopack's persistent cache barely helped.

The cache is per-bundler. My reset was fine; my ordering was not. Once I primed each bundler against its own cache before timing it, the real 143 millisecond warm compile appeared. The lesson is the same one as always: when a result looks flat, suspect the harness before the thing you are measuring. A warm build that is secretly cold looks exactly like a cache that does not work.

Run it yourself

Scaffold an app, then the comparison is two flags:

npx create-next-app@latest app --ts --app
cd app
rm -rf .next && time npx next build --webpack     # old default
rm -rf .next && time npx next build --turbopack   # new default

To see the phase breakdown that matters, timestamp each line and watch for the build-traces gap:

rm -rf .next
start=$(date +%s.%N)
npx next build --webpack 2>&1 | while IFS= read -r l; do
  printf '[%5.1fs] %s\n' "$(echo "$(date +%s.%N) - $start" | bc)" "$l"
done

The difference is small on a starter template and grows with your route count, so measure your own app, not a fresh one. I ran mine on an ordinary Linux laptop with Node 24.

What to do about it

If you are on Next.js 16 the default already gives you this, so there is nothing to switch on. The thing to change is what you expect and what you cache. Cache .next/cache in CI, because the warm compile is where the real speedup lives and a cold build throws it away. Do not expect the total build time to fall as far as the bundler benchmark suggests, because on a warm build most of your time is type checking and static generation, and those are exactly as slow as they were. And if your CI ever runs both bundlers, give them separate caches or the fallback to cold will be silent.

The 2-to-5x number is not wrong. It is just answering a smaller question than the one you have, which is how long you wait for the whole build. On this app that answer was 4x, most of it from a phase the benchmarks do not name.

I rented three servers, told nobody, and watched the internet break in anyway

Remdore — Fri, 11 Sep 2026 05:46:00 +0000

I rented three servers this afternoon, one in Frankfurt, one in New York, one in Singapore. I gave the addresses to nobody. They are not in DNS, not in a certificate, not committed anywhere, not pasted into a chat. The only systems that knew these machines existed were the ones I rented them from.

Frankfurt got its first uninvited connection 107 seconds after it booted. Singapore took 48. Over the next three quarters of an hour the three of them logged 3,480 connection attempts from 860 different addresses, and 1,085 attempts to log in with a real password.

None of this is aimed at me. Nobody chose these machines. This is just the background radiation of a public IP, and I wanted to measure it rather than wave at it.

The setup

Each box ran a small Python program that listens on the ports things actually knock on: telnet on 23, HTTP on 80 and 8080, TLS on 443. The SSH honeypot sat on 2222 so the machine's real SSH could keep port 22 for me to administer it.

Every listener is a fake. The telnet service shows a Ubuntu login prompt and rejects every password. The HTTP one logs the request and returns 404. Nothing is ever let in, and none of the credentials collected are used for anything. Alongside the listeners, tcpdump recorded every inbound TCP SYN that was not a reply to my own traffic, so I could count knocks on ports where nothing was listening as well.

The whole thing ran for about 45 minutes per region and then destroyed itself. Total cost was under three cents.

First contact is fast

Region	First unsolicited packet
Singapore	48 seconds after boot
Frankfurt	107 seconds
New York	324 seconds

There is no grace period. The address space is scanned continuously by everyone at once, so the clock starts the moment the IP is routable, not when you deploy anything. Whatever you were planning to harden after the demo works is already being probed.

The volume is steadier than the totals suggest

The raw totals are lopsided. Frankfurt logged 2,325 attempts against Singapore's 516, which looks like a big regional difference until you look at who sent them. A single host on Korea Telecom's network sent 1,663 packets to Frankfurt on its own, almost all to telnet. It was hammering one box, not scanning.

Take that one address out and the three regions agree closely:

Region	Attempts/hour, busiest source removed
Frankfurt	829
New York	759
Singapore	677

That is the real number I would quote: a fresh public IP, anywhere, fields somewhere around 700 to 800 unsolicited connection attempts an hour from the general noise, plus the occasional individual who decides to lean on you.

The shape of the crowd backs this up. Of the 860 distinct addresses, 672 of them, 78%, sent exactly one packet in the whole window. This is overwhelmingly wide, shallow scanning: touch an address, note what answers, move on. Only a handful of hosts ever settled in to actually try things.

What they want

Ranked by connection attempts across all three boxes, the ports asked for most:

23, telnet. By a mile, once the Korea Telecom host is included. This is the IoT story. Telnet is where routers, cameras and DVRs live, and the botnets never left.
8088. Steady on every box, from many sources. That is the Hadoop YARN and various admin panels port. Someone maintains a list.
22, SSH. Constant but not dramatic.
587 and 465. On New York specifically, one host tried the mail submission ports 50 times each. Freshly rented cloud IPs are prized for sending spam before anyone has reported them.
5555. Android Debug Bridge. A phone or TV box with ADB exposed is a free computer.

The HTTP requests were almost all bare probes to / with no Host header, which is a scanner checking whether anything answers rather than a browser. Only a trickle carried a real hostname, and none of them matched anything a previous tenant of these IPs might have run.

The passwords are the interesting part

The telnet listener collected 1,085 login attempts with an actual password, covering 107 distinct username and password pairs. This very nearly did not work, and the reason it failed at first is the best thing I learned all day. A bot connects, sends the username, and then waits for the server to negotiate telnet options before it sends the password. My first prompt did not speak that negotiation, so I collected a pile of usernames and no passwords at all. The fix was to answer the option negotiation the way a real telnet daemon does. Then the passwords poured in.

Here are the most common, by frequency:

Password	Times tried
annie2015	43
OxhlwSG8	42
S2fGqNFs	40
annie2014	40
annie2016	39
motorola	39
tlJwpbo6	39
7ujMko0admin	37
epicrouter	34
123456	33

If you have ever looked at a Mirai source dump these will be familiar. 7ujMko0admin is a Dahua camera backdoor. epicrouter is a Conexant DSL default. Zte521, further down the list, is a ZTE router default. The annie20xx run and the random eight-character strings like OxhlwSG8 are a specific botnet family cycling its built-in list. 123456 is in there too, because it always is.

The thing to notice is what is absent. Almost nobody tried to guess a human's password. There were no dictionary runs against admin with password1, qwerty, letmein. The traffic is not trying to break into accounts. It is trying every known factory default for every mass-produced device, at machine speed, against every address at once. You are not being attacked as a person. You are being enumerated as a device that might have shipped with a known password.

What I got wrong on the way

Three things, and the first two cost me real money, though not much of it.

I tried to move the machine's real SSH off port 22 so the honeypot could have it, by editing sshd_config. On Ubuntu 24.04 that does nothing, because SSH is socket-activated by systemd and the config's Port line is ignored. My attempt to fix that through the systemd socket instead took the box off the network completely, and I had to destroy it and start over. I did it a second time on another box before I accepted that relocating the admin SSH was not worth the risk and left it on 22, with the honeypot's SSH on 2222. The teardown ran in a finally block throughout, so both dead boxes were destroyed within seconds rather than billing overnight.

I also nearly reported a beautiful, clean finding of zero. My TLS parser pulled the server name out of every handshake and returned nothing, every time, and I almost wrote "nobody sends SNI to a bare IP". The parser was broken: it ended in a .decode('idna') that throws on an ordinary hostname, and a broad except turned every exception into a tidy None. A result of zero that comes from a bug looks exactly like a result of zero that is true. The only defence is to test your instrument against a positive control, so I pointed a real client with a known hostname at it and watched the None come back when it should not have.

Run it yourself

The listeners are plain sockets. The telnet negotiation is the one non-obvious part, and it is the difference between collecting passwords and collecting nothing:

IAC = 0xff
def handle(conn):
    # DO suppress-go-ahead, WILL echo, WILL suppress-go-ahead
    conn.sendall(bytes([IAC, 0xfd, 3, IAC, 0xfb, 1, IAC, 0xfb, 3]))
    conn.sendall(b'\r\nUbuntu 24.04.1 LTS\r\nlogin: ')
    user = read_line(conn)          # strip IAC sequences from the reply
    conn.sendall(b'Password: ')
    pw = read_line(conn)
    log(user=user, password=pw)     # then always reject
    conn.sendall(b'\r\nLogin incorrect\r\n')

The connection count came from tcpdump rather than the listeners, because I wanted to see knocks on ports where nothing answered too. This is where I burned an hour the first time I did this sort of thing, so it is worth saying plainly. If you capture everything you will mostly record your own machine talking back to apt and pip, and the number comes out enormous and meaningless. You want inbound SYN packets only, the opening knock of a connection and nothing else, and you want to exclude your own address so the SSH session you are watching from does not count itself. That is this:

tcpdump -i eth0 -nn -q \
  "tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0 and not host YOUR_IP" \
  -w knocks.pcap

Mine were the cheapest droplets DigitalOcean rents, one in Frankfurt, one in New York, one in Singapore, each gone the second its capture ended. I did not use three regions to compare them against each other. I used three because I wanted to know if this was some quirk of one datacentre or just what the internet is like everywhere, and it is everywhere: the knocking started within a couple of minutes on all three, wherever they were.

What to do about it

What stayed with me is how dumb all of it is. Nothing tried to be clever. It was 860 machines reading down the same list of factory passwords for cameras and routers, firing it at every address they could reach, and my three boxes were simply in the range that afternoon. If your server has no default credentials and nothing extra listening, this traffic does not get defeated, it just misses, because it was never looking for you in the first place.

So the advice everyone already gives turns out to be right, which is not surprising, except now I have watched why. Keep telnet off the internet. Change the password that shipped with the box, especially on the cameras and routers where it never gets changed because nobody signs back into a camera once it is on the wall. And the idea that you deploy first and harden afterwards does not survive contact with the numbers here, because Singapore was being probed 48 seconds in. The afterwards you were counting on is already gone.

nginx will proxy the new HTTP QUERY method. It will never cache one.

Remdore — Thu, 10 Sep 2026 08:36:00 +0000

The IETF published RFC 10008 back in June, and it adds a method to HTTP called QUERY. The short version is that it's a GET that's allowed to carry a body, so it's safe and idempotent like a GET, and it's cacheable, with the extra rule that whatever you put in the body has to end up in the cache key. The use case is the read whose parameters won't fit sensibly in a URL. Right now we all send those as POST and quietly accept that nothing downstream will ever cache them.

I've been curious whether any of the surrounding machinery is ready for it, so I spent a morning finding out. I wrote a search API that refuses every method except QUERY, put it on a small box in Frankfurt, and pointed ten language models at it to see whether they could produce a working client.

My assumption going in was that the models would be the weak link. That turned out to be wrong, and the thing that actually broke was the proxy.

The API

It searches the RFC index, which seemed like a fair dataset to pick given the circumstances. There are 9,836 entries in it. The filter document is exactly the shape RFC 10008 describes: optional substring matches on title and author, a year range, a list of acceptable statuses, a sort key, a limit.

Send it a GET or a POST and you get a 405 back, with an Allow: QUERY header and a body that names whichever method you tried. That rigidity is deliberate, because a server willing to also accept POST would have quietly absorbed every interesting failure instead of showing it to me.

$ printf '%s' '{"title_contains":"QUERY","limit":1}' \
  | curl -s -X QUERY --data-binary @- -H 'Content-Type: application/json' http://.../rfcs
{
  "total_matched": 21,
  "returned": 1,
  "results": [
    { "id": "RFC10008", "title": "The HTTP QUERY Method", "year": 2026, ... }
  ]
}

When the prompt names the method, everything works

My first prompt was explicit about it: this endpoint takes one method, QUERY, from RFC 10008. Each model got a URL path of its own so that the server's log could tell me who sent what, and I ran every generated program rather than reading it and forming an opinion.

Eight models were reachable on my account. Three families came back 403 on this tier and one returned a 400, which is a lesson about inference platforms rather than about HTTP, so I left it alone.

Every one of the eight produced a program that sent a QUERY and printed the five RFCs I'd asked for. Nobody hedged, nobody fell back to POST. Here's the log:

QUERY /rfcs/llama-4-maverick        QUERY /rfcs/deepseek-v4-pro
QUERY /rfcs/mistral-3-14b           QUERY /rfcs/gemma-4-31b-it
QUERY /rfcs/glm-5-3-flash           QUERY /rfcs/openai-gpt-oss-120b
QUERY /rfcs/openai-gpt-oss-20b      QUERY /rfcs/minimax-m2-5

Python helps here. urllib.request.Request(url, data=body, method='QUERY') is the whole trick, the standard library has never cared what you put in that argument, and all eight found it without visible effort.

When it doesn't, nobody guesses right

The version I actually wanted was the one where the method never comes up. So I rewrote the prompt the way a colleague would throw it at you in passing: here's a URL, here are the fields, it takes JSON, write me a client.

All eight sent POST.

None of them tried an OPTIONS request first. None of them treated the method as an open question at all, which is reasonable, because a JSON search endpoint has been POST for fifteen years and that prior is earned.

Two were cleverer than the rest about what happened next. deepseek-v4-pro and openai-gpt-oss-20b had both written code that caught the 405, looked at the Allow header while running, and retried using whatever it named. Neither was asked to. You can watch it in the log:

/r1/deepseek-v4-pro        POST -> QUERY
/r1/openai-gpt-oss-20b     POST -> QUERY

Given the answer, half of them take it

Next I pasted each model the exact failure it had caused: status line, headers with Allow: QUERY, OPTIONS among them, and the JSON body spelling out the method it should have used. Fix your program.

Model	What it did with the 405
deepseek-v4-pro	switched to QUERY, worked
gemma-4-31B-it	switched to QUERY, worked
openai-gpt-oss-120b	switched to QUERY, worked
openai-gpt-oss-20b	switched to QUERY, worked
llama-4-maverick	used QUERY, moved the filters into the URL, sent no body, got 400
mistral-3-14B	switched to GET, got another 405
glm-5.3-flash	emitted a fragment that wouldn't parse
minimax-m2.5	emitted a syntax error

Four out of eight, from a response that hands over the answer in a header.

The llama attempt is the one I keep thinking about, because it changed the method correctly and then url-encoded all the filters into the query string and sent an empty body, which throws away the single reason the method exists. Mistral's is blunter. The server said Allow: QUERY and it went with GET, which the same server had refused thirty seconds earlier.

nginx will carry a QUERY, and will never cache one

This matters more than anything the models did.

Proxying is fine. Drop nginx in front of the API and every request goes through untouched. The trouble starts when you try to cache, which RFC 10008 explicitly permits:

$ nginx -t
nginx: [emerg] invalid value "QUERY" in /etc/nginx/sites-enabled/default:9
nginx: configuration file /etc/nginx/nginx.conf test failed

proxy_cache_methods takes GET, HEAD and POST. You can't extend the list, and asking for anything else stops the server from starting.

The workaround anyone would try next is rewriting the method upstream with proxy_method POST and declaring POST cacheable. That fails too, and the reason is subtle: nginx tests proxy_cache_methods against what the client sent, not what it forwards. I checked by counting the requests that actually arrived at the backend, holding the config and the body constant and varying only the method on the wire.

What the client sent	Client requests	Requests that reached the backend
QUERY	4	4
POST	4	1

Sent as POST, three of the four never got past nginx. Sent as QUERY, the cache sat there doing nothing, and there isn't even a cache status header to hint that it declined.

Which leaves us somewhere silly. The method designed to be cached can't be, by the proxy most of the internet runs, while the method that isn't supposed to be cacheable is.

What I got wrong on the way

Three things, and I nearly published the second one as a result.

My server read Content-Length and stopped there. Node's http module doesn't set Content-Length for a method it has no opinion about, and reaches for Transfer-Encoding: chunked instead, so my server found an empty body, replied anyway, and left the chunks sitting unread in the socket. Whatever came next on that keep-alive connection parsed as rubbish, and node told me Parse Error: Expected HTTP/, RTSP/ or ICE/. I lost twenty minutes to believing node couldn't send a QUERY. It can. If you write one of these servers, handle chunked bodies.

To work out which method each program used, I grepped its source for method names. That reported three models picking GET in the no-hint round. The server log said all eight sent POST, every time. My regex had matched the word inside a comment or an unused error branch. The figure in this post comes from the log, because the log is a record of what happened and my grep was a record of what I assumed.

Then there's my own Content-Location header, which advertises /rfcs?limit=1&title_contains=QUERY. RFC 10008 offers that header as the GET-able equivalent a cache can hold onto. Mine answers 405 to a GET, so it's pointing at a door that doesn't open. I only spotted it while checking something unrelated.

Run it yourself

The server is roughly a hundred lines of standard library Python. The client is four:

import json, urllib.request
body = json.dumps({"title_contains": "HTTP", "year_from": 2020, "limit": 5}).encode()
req = urllib.request.Request(URL, data=body, method="QUERY",
                             headers={"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req))["total_matched"])

For the nginx behaviour, put any backend behind proxy_cache with proxy_cache_methods POST, send one body four times as POST and four times as QUERY, and count arrivals at the backend instead of reading headers:

for i in 1 2 3 4; do curl -s -o /dev/null -X QUERY --data-binary '{"a":1}' http://localhost/; done
for i in 1 2 3 4; do curl -s -o /dev/null -X POST  --data-binary '{"a":1}' http://localhost/; done

I reached the models through DigitalOcean's inference endpoint, and the API sat on a 1-vCPU droplet in Frankfurt, so nothing in the model experiment went over loopback.

What to do about it

Clients are ready. Python, curl and node all send QUERY today without any ceremony, and the models writing your integration code will get it right as soon as your documentation mentions it. Don't skip that step. Eight out of eight assumed POST, and the only two that arrived at QUERY on their own got there by reading a 405 mid-flight.

It's the middle of the stack that will disappoint you. Count what reaches your backend before believing anything about caching, because nginx fails at this silently, with no error and no header, just every request sailing through. I only noticed because I gave up on headers and started counting.

And if caching is your reason for wanting QUERY at all, look at your proxy before you commit to the protocol. On nginx as it stands you can keep sending POST and cache it, or you can put something else in front. Neither is what the RFC had in mind, and a method nothing will cache is POST with better manners.

DEV Community: Remdore

92% of dev.to posts get zero reactions, and the title advice does not change that

Two websites wearing the same logo

Getting a fair sample is the hard part

What actually correlates with anything

The part that undoes most of the above

Where I sit, since it would be cowardly not to say

What I got wrong

What to take from it

Only 10 of the top 1000 sites hide the hostname you are visiting. I built the tool that counts.

What it actually does

So I pointed it at the top thousand sites

The number that is not the number

Where it ran, and where it is now

What I got wrong on the way

Contributing

The takeaway

The hostname you visit over HTTPS is sent in plain text

The hostname, sitting in the packet

Why it is sent in the clear

A passive listener needs no key and no handshake

TLS 1.3 does not fix it

The actual fix, and where it stands

What I got wrong on the way

See it yourself

What to take away

A container is just a process with a few private views. I built one in thirty lines.

Namespaces are the whole trick

Swapping the root filesystem is the fiddly part

The mistake that made ps lie

Namespaces isolate. Cgroups limit.

What it is not

Run it yourself

What to take away

ESLint took 4.4s to lint Vue's core. oxlint took 0.24s. Then I turned on the type-aware rules.

The setup

The syntactic number is 18x, and that undersells it

Type-aware is where I was wrong

The honest caveats, because there are several

Where a football site feels this

What I got wrong on the way

Run it yourself

What to do about it

A warm pip install took me 13 seconds. uv took 56 milliseconds.

What I measured

The warm number is the one that matters, and it needs explaining

The catch, and it is a real one

Where a football site would feel it

What I got wrong on the way, and a real aside

Run it yourself

What to do about it

I misspelt 12 footballers' names. Postgres full-text found 4, Meilisearch found all 12.

The setup

Full-text search does not tolerate a single mistake

Trigram is better, and that is where it gets dangerous

Meilisearch got all twelve, and was faster

Where this actually matters

The honest cost

What I got wrong on the way

Run it yourself

What to do about it

My benchmark harness was wrong fourteen ways before it measured anything

The worst one was invisible in the output

The guard had a tenth of the power it appeared to have

It blamed a proxy for its own stalls

It perturbed the thing it was measuring

The metric's core logic had no test at all

It undercounted frames, silently

The guard trusted the log it was auditing

The self-test certified a server it had not started

Two smaller ones, same shape

Then I built a mechanism that did not exist

What actually made the difference

The uncomfortable part

nginx streams your tokens fine. HAProxy holds them for 206ms.

The metric

Token-sized frames are HAProxy's worst case

The header everyone reaches for does nothing

When proxy_buffering does cost you something

The finding I killed

When `proxy_buffering` does cost you something