When I sat down this morning my problem was small: my blog's JSON feed was going out uncompressed and I meant to find out why and fix it. I found out why. In fact I found two reasons. Then I asked "so how much is this costing how many people", looked at the bill, and discovered that the defect I had been chasing was the minor half of the story. Almost all of my feed traffic is produced not by readers, but by me.
The thesis of this piece fits in one sentence: diagnosing a configuration defect is easy, measuring who pays for it is hard, and because I never did the second one, I spent seven months looking at the wrong side of it.
What I was looking for: a feed going out bare
The blog has two feeds. rss.xml is classic RSS, feed.json is JSON Feed. Same content, two formats. I added a random query string to each URL to bypass the cache and requested both with five different Accept-Encoding headers:
feed.json AE=identity -> enc=NONE 1,434,299 bytes
feed.json AE=gzip -> enc=NONE 1,434,299 bytes
feed.json AE=br -> enc=NONE 1,434,299 bytes
feed.json AE=zstd -> enc=NONE 1,434,299 bytes
feed.json AE=gzip, br, zstd -> enc=NONE 1,434,299 bytes
rss.xml AE=identity -> enc=NONE 1,921,059 bytes
rss.xml AE=gzip -> enc=gzip 270,024 bytes
rss.xml AE=br -> enc=br 256,723 bytes
rss.xml AE=zstd -> enc=zstd 261,418 bytes
rss.xml AE=gzip, br, zstd -> enc=zstd 261,418 bytes
I read that table twice the first time. In raw form the RSS feed is larger than the JSON feed — 1,921,059 against 1,434,299 bytes — yet it travels as 261 KB. The JSON feed, no matter what I ask for or which algorithms I offer, arrives with all 1.4 MB of itself. The smaller one travels heavier.
My first suspicion was my own code, and it turned out to be justified for the wrong reason. src/pages/feed.json.ts builds the feed with JSON.stringify(feed, null, 2) — two-space indentation, so a human can read it. I stripped the indentation and measured: 1,233,489 bytes. A 200,810-byte difference in raw size, 14 percent. For a moment I thought I had the culprit and relaxed, then I compared the compressed versions — 166,618 bytes indented, 163,362 stripped. A compressor swallows repeated whitespace almost for free. The thing I wanted to blame is a two-percent matter; the real matter is that compression never runs at all.
The difference sits in one line: the media type
When I put the two feeds side by side, the only meaningful difference in the response headers was this:
feed.json -> content-type: application/feed+json; charset=utf-8 (NO compression)
rss.xml -> content-type: application/xml (zstd)
The site sits behind Cloudflare, and Cloudflare makes the compression decision. I read its documentation: on the free plan content is compressed with Zstandard by default, but only for the content types the documentation enumerates one by one. Reading that list was instructive. It contains application/rss+xml. It contains application/ld+json, application/manifest+json, application/geo+json, application/graphql+json, application/vnd.api+json, application/xhtml+xml, image/svg+xml — the suffixed types are enumerated one at a time, each of them a row somebody typed. application/feed+json is not there. Plain application/json is on the list too, which is why /manifest.json on my own site arrives compressed, as the same test showed.
Here is the thing: the +json suffix is not a casual naming habit, it is a registered structured syntax suffix. RFC 6839 defines it precisely so that a recipient can do "generic processing of the underlying representation" — that is, so middleware seeing application/whatever+json can assume the payload is JSON even if it knows nothing about what the type means. A compression layer is the textbook case: the compressor does not need to know what the feed means, only that it is text. But the list does not match the suffix, it matches the string. The suffix does nothing unless someone writes it into the list.
That also points at the cheapest remedy: changing the feed's Content-Type to application/json is a one-line edit, and that type is already on the list. The price is giving up the type JSON Feed recommends — sacrificing a meaningful signal to feed readers for the sake of a compression table. It is not my preference, but I would rather not hide how simple the option is.
I checked the IANA registry myself too — downloaded the full set of application types and searched all 1,824 rows. feed+json is not registered. What I expected here was a tidy explanation: "so it never made the list because it was never registered." Not so. rss+xml is not registered either, graphql+json is not registered either, and both sit on the list. Registration is not the criterion. The criterion is that someone thought to type that row.
I checked the origin side as well, because Cloudflare requests content from the origin with accept-encoding: br, gzip; had the origin compressed, the edge could have passed it through. The blog's vhost file does not contain a single gzip or brotli line, and since gzip can be inherited from the http block I checked the global file too — the only gzip line in /etc/nginx/nginx.conf sits inside a comment (#gzip on;) and the directive appears nowhere under conf.d. In nginx the gzip default is off, so the origin compresses nothing. The lightness of my site's pages is not my work; it is the work of Cloudflare's default. Writing that down stung a little.
I found the same omission in my own file
Something else was missing from the feed response: there is no Cache-Control header at all. Yet the application code sets one, public, max-age=3600. I found where it goes in the nginx configuration:
map "$mustafa_blog_html_no_store|$sent_http_content_type" $mustafa_cache_control_override {
default "";
"~*^1\|text/html" "no-store, no-cache, must-revalidate";
"~*^0\|text/html" "public, max-age=0, s-maxage=180, stale-while-revalidate=60";
"~*\|application/rss" "public, max-age=0, s-maxage=900";
"~*\|application/atom" "public, max-age=0, s-maxage=900";
"~*\|application/xml" "public, max-age=0, s-maxage=900";
}
Right below it, the location / block hides the application's header and substitutes whatever this map returns. The map knows application/rss, application/atom, application/xml. Three dialects of XML. It does not know the JSON feed's type, so the match falls through to default "", and a header added with an empty value is not added at all. Result: the retention my application asks for reaches nobody, and the edge decides caching by its own default.
This is a file I wrote. While taking issue with Cloudflare's list, I had made the same mistake for the same reason with my own hands: I kept a hand-enumerated list of media types and forgot to write down the JSON feed — the same habit in two separate layers.
The cost of the missing header is not merely on paper either. When I split the feed's 49 requests over the last 24 hours by cache status, 16 were warm hits, 18 were expired and 15 were outright misses. In other words only a third of the requests were served from cache: in 18 of them the edge went back to revalidate the copy it held, and in 15 it held no copy at all. Because the one-hour window my application offers is lost in transit, the cache does not work for me, it negotiates with me.
One more small inconsistency surfaced in the same dig: there is no /en/feed.json — it returns 404. I gave the English side RSS and never gave it a JSON Feed. I had not noticed because nobody asked for it, which could have been a subheading of this very piece.
Then I looked at the bill
I wanted to know how many people this affected. First I looked at my own server's access log — and found nothing. In a window of 17,796,181 lines stretching from 12 September to 3 November, the exact path /feed.json appears in only 31 requests, all of them 301 redirects. In 53 days the origin produced not one 200 for the feed. On top of that the log file is shared by every site on that server and the lines carry no host field, so I cannot attribute what I found to this site either. I cannot see the cost at the origin because the origin does not serve the requests; the edge does. That is precisely why the defect stayed invisible: the party paying the bill does not show me the bill.
So I asked Cloudflare's analytics API. Last 24 hours, responses that returned 200 only. A warning up front: these figures are estimated from sampled data, not counted byte by byte — I will show you below where that shows.
| Address | Requests | Total | Average per request |
|---|---|---|---|
/rss.xml |
5,374 | 9.53 GiB | 1,904,825 bytes |
/en/rss.xml |
2,681 | 4.75 GiB | 1,903,449 bytes |
/feed.json |
48 | 62.0 MiB | 1,353,350 bytes |
14.35 GiB in a single day. The address where my defect lives, feed.json, is the smallest row in that table: 48 requests and 62 MiB a day. I had spent the morning on the uncompressed feed; that feed is 0.4 percent of the total. The row next to it — the feed whose compression works — is 9.53 GiB.
And the averages sit in the wrong place. Compressed, the RSS feed drops to 261 KB, yet the average per request is 1,904,825 bytes. So compression works and most clients are not asking for it.
A measurement trap: what is this number counting?
Here I had to stop and decide whether to trust my own data. Does edgeResponseBytes count the body before compression, or the bytes that go on the wire? If it is the former, the 14.35 GiB above is not real traffic and the whole article is garbage.
I tested it against an address whose sizes I know. The home page is 122,887 bytes raw and 31,629 bytes with zstd. The average the edge reports for the same day is 95,632 bytes — neither of the two, but somewhere between them. So the number counts wire bytes, and the average is a mixture of compressed and uncompressed responses.
Those same two ends give me the ratio. On the home page the compressed share is 30 percent; on /blog/ (raw 154,445, compressed 35,197, average 43,599) it is 93 percent. The gap taught me something. I was about to take the shortcut — browsers ask for compression, so a high ratio means humans — and the home page refutes it. The home page is the address this site serves to more browsers than any other, and its compressed share is a third. Search engine crawlers negotiate compression, and so does most HTTP tooling; I have already measured that most of this blog's visitors are bots. The ratio does not tell me whether a client is human. It tells me whether it negotiated.
The same arithmetic applied to the RSS feed puts the compressed share at around one percent. These ratios are derived, not measured: three magnitudes came off the wire — raw size, compressed size, edge average — and the share comes from arithmetic on top of them. Let me name the weak spots too: the three algorithms' outputs differ by five percent and I assumed a single "compressed size"; publishing a post inside the window changes the feed's raw size; and what comes out is a byte-weighted share, not a request-weighted one. Still, the result has an independent check: in the client breakdown below, the clients that do ask for compression come to 60 of the 5,374 requests, or 1.1 percent. Two roads, one destination.
Even so, the distance between 30 percent and 1 percent wants an explanation. Something is pulling this feed without asking for compression in ninety-nine requests out of a hundred. I needed to ask what.
Who pays the bill
I broke it down by client signature. Twenty-four hours of the Turkish feed:
| Client | Requests | Average per request |
|---|---|---|
curl/8.5.0 |
2,815 | 1,921,459 bytes |
curl/8.18.0 |
1,254 | 1,921,650 bytes |
alert-monitor/1.0 (+https://mustafaerbay.com.tr health-check) |
1,245 | 1,921,641 bytes |
feedparser/6.0.11 |
11 | 272,094 bytes |
opr-poller/0.1.0 |
7 | 266,995 bytes |
| browsers (Firefox and Chrome) | 11 | ~274,000 bytes |
The first three rows account for 5,314 of the 5,374 requests, 98.9 percent, and all three take the feed raw. That leaves 60 requests; the table's last three rows — actual feed readers and humans — are 29 of them, every one around 270 KB, because every one negotiates compression. Of the remaining 31, the 21 I can see look the same: a feed library, two crawlers, and the eleven requests my own Mac made this morning while taking these measurements. Yes, I am on that list too; this article generated its own cost. On the English feed the picture is sharper: nearly all of the 2,681 requests come from two clients, the remainder is six requests, and those six are curl on my own laptop again.
This is where the sampling shows itself. The two clients come to 1,415 and 1,269, which adds up to 2,684 — nine more than the listed total. When I ask for the same data in two separate queries, Cloudflare estimates each from its own sample and the numbers do not reconcile. That does not change the order of magnitude, but it does forbid me from saying "I counted byte by byte": the figures below are the best estimate I have, not an exact census.
So: feed readers were not the problem. They never were. My own machines download a 1.9 MB XML file thousands of times a day.
curl/8.5.0 is the exact version of curl on my VPS3. curl/8.7.1 is my Mac's version. alert-monitor/1.0 carries the name of my own alerting service and points at my own domain in its identity string. I could not tie curl/8.18.0 to any of my machines in this pass, and I could not find the code that sends the alert-monitor/1.0 identity either — I searched /usr/local, /etc, /opt/mustafaerbay on the server and the job definitions on my Mac, with no result. I am leaving both as open items and will write about them once closed.
1,921,059 for three bytes
The part I could trace with the code in hand is the watchdog script on VPS3. It touches the feed in two places. The first is the health probe:
probe_url() {
local url="$1" code=000 attempt
for attempt in 1 2 3; do
code=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 6 --max-time 25 "$url" 2>/dev/null)
...
}
When I wrote -o /dev/null I thought I knew what I was doing: I do not want the body, I throw it away. And I do throw it away — after it has crossed the network. What curl refuses here is writing the full response to disk, not downloading it. The information I keep is %{http_code}: three characters, 200. For three bytes, 1,921,059 bytes. A ratio of 640,353.
The second one is even more elegant:
RECENT=$(curl -s ... "https://mustafaerbay.com.tr/rss.xml" | grep -oP "<link>[^<]+" | sed -n "2p" | ...)
To learn the address of the latest post it downloads the entire feed and takes the second <link> element. The value it got this morning was this: https://mustafaerbay.com.tr/blog/career/32-soz-verdim-denetleyebildigim-13unun-11i-acik/ — 88 bytes. I download 1.9 MB for 88 bytes that sit in the file's first few kilobytes, a ratio of 21,830.
The timer says OnUnitActiveSec=1min. It runs once a minute, and each run makes two requests to the Turkish feed and one to the English one. The expected daily counts are 2,880 and 1,440; the measured ones are 2,815 and 1,415 — within two percent. Given the sampling, two numbers landing that close is not coincidence: the traffic the edge reports is the traffic my own timer produces.
Across all the feeds together that is an average of 5.5 requests a minute, so the 1.9 MB in the title is what one probe costs; the minute's full bill is five times that.
The same information, four different prices
Here is the same probe, priced four ways:
as it is today (GET) : 1,921,059 bytes
HEAD (-I) : 0 bytes
GET --compressed : 274,610 bytes
GET -r 0-4095 (first 4 KB) : 1,921,059 bytes <- range ignored, http=200
For a status code HEAD suffices and no body arrives at all. But while I was considering HEAD, a note in my own configuration file stopped me: on 6 July 2026 I wrote that RSS could be served stale for up to a week, because on 29 June the live feed had been caught frozen. Send HEAD to a frozen feed and you get a 200; the watchdog reports health while the very fault it exists for passes under its nose. So HEAD zeroes the probe's cost and blinds it to its own reason for being. The right answer sits in the third row and I had already measured it: asked with If-Modified-Since, an unchanged feed returns 304 and zero bytes, and a changed one returns the body — free, and the only option that can see staleness. If I genuinely need to read the content, --compressed cuts it sevenfold; curl does not send Accept-Encoding by default, so I get no compression because I never asked for it. The range request does not help: the feed is a response generated on the fly, it advertises no Accept-Ranges, and the server ignores the range and hands over the full body. I went through this in detail recently: in the piece I published yesterday I showed that on this same site the gate for partial downloads is an Accept-Ranges announcement plus a Content-Length, and that without both the directive does not save you either. The same gate is shut here. I include this row not as a new finding but to show why the watchdog's fourth option is ruled out: a range is a promise the server makes about that resource, not a wish the client expresses.
By rough arithmetic — a calculation, not a measurement — my own clients' 7,998 daily requests come to 14.31 GiB today and would be 2.05 GiB if every one of them asked with --compressed: 12.26 GiB a day, 4.37 TiB a year. I left third parties' 60 requests and feed.json out of that sum; they are not mine to change. And in honesty not all of it is applicable today either: 2,514 of the 7,998 requests, 31 percent, belong to the alert-monitor/1.0 client whose code I could not find. My two flags reach two thirds of the traffic; for the last third I have to find that script first.
One more reading, out of curiosity: asked with an If-Modified-Since header, both feeds return 304 with a zero-byte body. So my server already gives the right answer to a well-behaved client. My watchdog does not ask.
Six questions for auditing your own tools
What this morning left me is a checklist, for my own monitoring:
- Does this probe genuinely need the body, or is the status code enough? If it is enough,
HEAD. - If the body is needed, am I asking for compression?
curldoes not ask; you have to say it. - Where I wrote
-o /dev/null, do I know what I saved? The disk, not the network. - Am I pulling a whole document for a single field? Does the document have a smaller form, an endpoint, or a conditional request?
- Who chose this probe's frequency, and against what? Once a minute — is that the rate at which the thing I measure changes, or just a round number?
- Who pays for the traffic? If a free layer absorbs it, that layer will not be showing me the bill.
I lingered on the fourth question, because in my case a smaller form really does exist: the JSON feed is smaller than the RSS one, and the latest post's address sits at the top of it as well. But the watchdog chose XML, and I never questioned it.
What remains
When I started writing this piece I had a clean story in hand: a provider's hand-kept list misses a media type, and I lose compression. The story is true, the defect is real, it sits in two places at once, and it will be fixed. But that defect belongs to an address that moves 62 MiB a day. Next to it, at the address where compression works flawlessly, my own watchdog pours out 14 GiB.
The difference between the two is not technical. This asymmetry feels familiar to me, because the habit of the profession runs this way. Reading a configuration file and finding a defect is pleasant work, because it has an end and you can show what you found. Counting who pays how much is tedious; it makes you build instrumentation first, and the number it produces usually indicts you. I did not even have to choose between the two — the first came to me on its own, and nobody asked for the second.
The defect was sitting in a readable place: in a configuration file, on a documentation page, visible to anyone who looked. The cost was written in no file at all; to see it you had to count who downloaded how much, and that number was never shown to me because a free edge layer was quietly swallowing the bill. What is diagnosable and what is expensive are not the same thing. The application/feed+json line went into the repository on 1 April 2026 and has not changed since, and the second defect, the one in the nginx map, has been there since 6 July. One is seven months old, the other four. How long the watchdog has been downloading once a minute I did not measure — so I know the cost's present rate, not its duration.
I have measured and written before that most of this blog's visitor counts are bots. Admitting the traffic was not real upset me in that piece. This time it is worse: 98.9 percent of the traffic is real, it just belongs to me. The tool I set up to watch my site has become the site's heaviest visitor.
Three weeks ago I audited my watchdogs too, and wrote that CI had run 1,973 times in 26 days purely to check health. What I counted there was run counts, and the question was whether those runs catch anything; here I counted bytes, and the question was what the probes cost. Two faces of the same habit, and I only saw either by sitting down to count.
Let me note the next question here as well, because I do not know its answer: the edge absorbs this traffic today. If I add a rule that changes caching behaviour, eight thousand feed renders a day could land on the origin, and the bill could move from free to my own server's CPU. I will not make the fix without measuring that.
Two items stay open and I am writing them down here so I do not forget tomorrow: find the code that sends the alert-monitor/1.0 identity and move it to HEAD, and work out which of my machines curl/8.18.0 is. I will add the compression rule too — but now I know which one comes first.
Official Sources
- Cloudflare — Compression (the compressible content type list and plan defaults)
- Cloudflare — Compression Rules (changing compression by media type)
- RFC 6839 — Additional Media Type Structured Syntax Suffixes (the definition of the
+jsonsuffix) - RFC 9110 — HTTP Semantics (the HEAD method and content negotiation)
- nginx — ngx_http_gzip_module (the
gzipandgzip_typesdefaults)
Top comments (4)
bro can you pitch me burn cpu?
Sure, bro! 😄
BurnCPU is an independent social platform I’m building with a focus on privacy, performance, and giving users more control over their experience.
The backend runs on Rust (Axum), with SolidJS on the frontend, PostgreSQL, Redis, and Meilisearch behind the scenes.
The idea is simple: build a fast, lightweight social platform without unnecessary complexity, intrusive tracking, or forcing users into an ecosystem they can’t leave.
I’m developing and hosting the infrastructure myself, so it’s also a real-world experiment in building a social network from the ground up.
Still evolving, but that’s part of the fun. 🔥
Would love to hear what you’d expect from a platform like this!
unique concept I would said for 2026 social platforms
lol we build what others forgot to fix
but its in development stage right? not active yet...?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.