DEV Community

Cover image for I Left a Stale .gz and Nginx 1.31.7 Served Two Bodies at One URL
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

I Left a Stale .gz and Nginx 1.31.7 Served Two Bodies at One URL

This blog's origin config has carried gzip on for years: the same CSS gets recompressed on every request. The well-known way to make that cheaper is obvious enough — compress the file once, drop it next to the original as .gz, write gzip_static on, and let nginx ship what is already there. Before flipping that switch, I set the mechanism up on my own machine, because what I wanted to know was not the gain. It was the price.

The price fits in one sentence. When nginx serves a precompressed file, it never looks at the original. And because it never looks, it can never learn that the .gz next door has gone stale. I ran that as a measurement, and the result came out sharper than I expected: one address, two different bodies — and which one you get depends solely on your Accept-Encoding header.

The setup: five server blocks, five questions

I installed nginx from Homebrew on my Mac. The version is the freshest mainline there is as I write this:

$ nginx -v
nginx version: nginx/1.31.7
$ nginx -V 2>&1 | tr ' ' '\n' | grep gzip_static
--with-http_gzip_static_module
Enter fullscreen mode Exit fullscreen mode

1.31.7 is the top entry in nginx's own CHANGES file, dated 14 October 2026 — so I am measuring a release that is two days old. The module itself has been sitting still for years: the last behavioural change recorded against ngx_http_gzip_static_module in the changelog is byte-range support in 1.23.4. This is not an old version's quirk, then. It is today's design.

The trick in the config is that I deliberately turn off on-the-fly compression and push the thresholds to absurd heights:

gzip off;                      # on-the-fly compression OFF
gzip_min_length 100000;        # do not compress anything under 100 KB
gzip_comp_level 1;
gzip_types text/plain;         # text/html is NOT in the list
gzip_vary on;

server { listen 127.0.0.1:8091; root .../www; gzip_static on; }
server { listen 127.0.0.1:8092; root .../www; gzip_static always; }
server { listen 127.0.0.1:8093; root .../www; gzip_static off; }
server { listen 127.0.0.1:8094; root .../www; gzip_static on;
         open_file_cache max=100 inactive=60s;
         open_file_cache_valid 30s; open_file_cache_min_uses 1;
         open_file_cache_errors on; }
server { listen 127.0.0.1:8095; root .../www; gzip_static on; gzip_vary off; }
Enter fullscreen mode Exit fullscreen mode

I put two cases in the docroot: a stale companion file, and a .gz with no original at all.

$ ls -lT www/a.html www/a.html.gz www/b.html.gz
-rw-r--r--@  19  Oct 16 08:47:59 2026  www/a.html      # "SURUM-2 taze govde"  (fresh)
-rw-r--r--@  40  Oct 15 08:00:00 2026  www/a.html.gz   # "SURUM-1 bayat govde" (stale)
-rw-r--r--@  36  Oct 16 08:47:59 2026  www/b.html.gz   # NO original
Enter fullscreen mode Exit fullscreen mode

First measurement: one address, two bodies

I fetched a single URL as two different clients. The second request sends no Accept-Encoding header at all — in curl, -H 'Accept-Encoding:' does not send an empty value, it removes the header entirely:

$ curl -s --compressed http://127.0.0.1:8091/a.html
SURUM-1 bayat govde
$ curl -s -H 'Accept-Encoding:' http://127.0.0.1:8091/a.html
SURUM-2 taze govde
$ cat www/a.html
SURUM-2 taze govde
Enter fullscreen mode Exit fullscreen mode

The real content on disk is SURUM-2. Your browser — that is, every client that asks for gzip — gets yesterday's body. The headers make it even clearer:

Header gzip-capable client client sending no Accept-Encoding
Content-Length 40 19
Content-Encoding gzip (none)
Last-Modified Thu, 15 Oct 2026 05:00:00 GMT Fri, 16 Oct 2026 05:47:59 GMT
ETag "6ad05dd0-28" "6ad1ba8f-13"
Vary Accept-Encoding Accept-Encoding

Nginx reports the size and the date of the file whose body it is sending. The .gz is a day old, so Last-Modified is a day old too. The ETag decodes as well: 0x6ad05dd0 is 1792040400, that is 15 October 08:00, and 0x28 is 40 bytes. So the ETag is derived straight from that .gz file's mtime and size — in the source the ETag format is "%xT-%xO", fed by last_modified_time and content_length_n (the ETag exists at all only because etag on is the default; turn it off and the line is never produced). Everything is consistent. Consistent with the stale file, not with reality.

The most informative row in that table is the last one: Vary is there in the non-gzip response too. I'll come back to it shortly, because that header is what gives away when the module actually runs.

Why nginx does not notice: the single path in the code

This is not a bug, it is the module's design. In ngx_http_gzip_static_module.c at the 1.31.7 tag, the handler maps the URI to a filesystem path, appends .gz, and opens only that path. There is no stat of the original, no attempt to open it, and no mtime comparison.

Diagram

The order in that diagram is a claim too, so I kept it faithful to the code, with two branches simplified away: a URI ending in / makes the module back off at the very start, and a .gz that is not a regular file returns 404.

The gzip_vary branch on the right is the surprising part. When the client does not ask for gzip, the module does not back off immediately: if gzip_vary is on it still opens the .gz path, flags the response to carry Vary: Accept-Encoding, and only then declines. The early exit exists only when gzip_vary is off. Two blocks settle it — 8091 with gzip_vary on, 8095 with it off, neither request asking for gzip:

# 8091: gzip_vary on
HTTP/1.1 200 OK
Content-Length: 19
Vary: Accept-Encoding

# 8095: gzip_vary off
HTTP/1.1 200 OK
Content-Length: 19
Enter fullscreen mode Exit fullscreen mode

The body is the original file in both cases, but the first one carries Vary. That header is the proof that gzip_static ran even on a request that wanted no gzip. In practice it means this: with gzip_vary on, every request — asking for gzip or not — tries to open the .gz path, so the open_file_cache entry and the file descriptor are held for those requests too.

The official documentation papers over the real gap with a recommendation — it says it is recommended that the modification date and time of the original and compressed files be the same. A recommendation. Not a requirement, and not a check. The moment those two timestamps drift apart on disk, nothing warns you.

The stale copy that says "not modified"

You might pause here and think caches will refresh at some point anyway. The numbers say otherwise:

$ curl -s --compressed -D - \
    -H 'If-Modified-Since: Thu, 15 Oct 2026 05:00:00 GMT' \
    http://127.0.0.1:8091/a.html | head -1
HTTP/1.1 304 Not Modified

$ curl -s -H 'Accept-Encoding:' -D - \
    -H 'If-Modified-Since: Thu, 15 Oct 2026 05:00:00 GMT' \
    http://127.0.0.1:8091/a.html | head -1
HTTP/1.1 200 OK
Enter fullscreen mode Exit fullscreen mode

Same moment, same address, same date: to the gzip client it says "not modified", to the plain client it says "here, take the new one". Sending the stale ETag with If-None-Match gets the same:

$ curl -s --compressed -D - -H 'If-None-Match: "6ad05dd0-28"' \
    http://127.0.0.1:8091/a.html | grep -Ei 'HTTP/|ETag'
HTTP/1.1 304 Not Modified
ETag: "6ad05dd0-28"
Enter fullscreen mode Exit fullscreen mode

The stale copy, in other words, declares itself fresh, and every intermediate cache in front of it takes that word and keeps what it has. There is no self-healing here. This file goes away only when a human notices it and deletes it.

Put a CDN in the path and the picture tilts further: thanks to Vary: Accept-Encoding the two representations are stored separately, one right and one wrong, and both are "fresh". No bug report arrives either, because the complaining user reloads the page, the server says 304 again, and the problem degrades into "it doesn't show up for me".

An address that exists only for gzip clients

If the handler never opens the original, what happens when the original does not exist? There is no b.html in the docroot, only b.html.gz:

$ curl -s --compressed -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8091/b.html
200
$ curl -s --compressed http://127.0.0.1:8091/b.html
yalnizca-gz-var
$ curl -s -H 'Accept-Encoding:' -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8091/b.html
404
Enter fullscreen mode Exit fullscreen mode

Same address, same second: 200 and 404. Whether the page exists depends on the client's header. In the block with gzip_static off, both are 404 — so the only thing bringing that address into existence is the module itself:

# 8093, gzip_static off
gzip client  : 404
plain client : 404
Enter fullscreen mode Exit fullscreen mode

This turns into a trap on the monitoring side. If your uptime checker does not send Accept-Encoding, it sees 404 while browsers see 200 — or the other way around, the checker goes green and the user tries to describe a page they cannot find. Monitoring and users living in different worlds is the most tiring kind of debugging there is.

Four knobs I had tuned do nothing, the other four still work

Remember the lab config: gzip off, gzip_min_length 100000, gzip_types text/plain. All three should have blocked a 40-byte .gz of a text/html file. None did:

$ curl -s --compressed -o /dev/null \
    -w '%{size_download} bytes, enc=%header{content-encoding}\n' \
    http://127.0.0.1:8091/a.html
40 bytes, enc=gzip
Enter fullscreen mode Exit fullscreen mode

Which makes sense, because nginx is not compressing anything here — it is shipping a ready-made file. gzip_comp_level is meaningless for the same reason: the compression level is now chosen by whoever runs the gzip command, which is you. The documentation's list of "the following directives are also taken into account" is short and clear: gzip_http_version, gzip_proxied, gzip_disable, gzip_vary. Anything not on that list is quietly inert while gzip_static is on.

One of the directives on that list does take effect, and here it is at work:

$ curl -s --http1.0 --compressed -D - http://127.0.0.1:8091/a.html | grep -i 'content-'
Content-Length: 19
Enter fullscreen mode Exit fullscreen mode

Requesting over HTTP/1.0 triggered the gzip_http_version 1.1 default: the module opened the .gz, flagged Vary, and backed off; the static module served the body — 19 bytes, the fresh one. A strange little irony: the shortest route to protection from the stale copy is for your client to speak an older protocol.

Knob While gzip_static is on
gzip on/off inert (measured)
gzip_min_length inert (measured)
gzip_types inert (measured)
gzip_comp_level inert (no compression happens)
gzip_http_version effective (measured)
gzip_proxied effective (docs and source)
gzip_disable effective (docs and source)
gzip_vary effective (measured)

One caveat matters here: by "inert" I mean inert for the response gzip_static sends. If another path on the same server is still compressed by nginx itself, every knob keeps working for that response. The two live side by side; what gets confusing is which response obeys which rule.

Opening my own config surprised me on exactly this point. The origin carries gzip on, gzip_proxied any, gzip_types ..., gzip_min_length 1024 — but every path, /_astro/ included, goes to proxy_pass http://127.0.0.1:3040. The only place that serves a file from disk is the 404 fallback pointing at the previous build. gzip_static is a static-file module; it never touches a proxied response. So in my configuration that line would change literally nothing — and all four directives keep full authority, because they are compressing proxied responses. gzip_proxied any is sitting there for exactly that reason.

There is also always. In that mode the module does not ask the client at all. Here you have to be careful, because not sending the header is not the same as saying you do not want gzip. With no header present, HTTP treats any content coding as acceptable, so gzip breaks no rule. The real test therefore needs a client that speaks up — Accept-Encoding: identity, meaning "give me the uncompressed one":

# 8091, gzip_static on
$ curl -s -H 'Accept-Encoding: identity' -D - http://127.0.0.1:8091/a.html \
    | grep -Ei 'HTTP/|Content-Length|Content-Encoding'
HTTP/1.1 200 OK
Content-Length: 19
$ curl -s -H 'Accept-Encoding: identity' http://127.0.0.1:8091/a.html
SURUM-2 taze govde

# 8092, gzip_static always
$ curl -s -H 'Accept-Encoding: identity' http://127.0.0.1:8092/a.html | xxd | head -1
00000000: 1f8b 0800 0000 0000 0203 0b0e 0d0a f5d5  ................
Enter fullscreen mode Exit fullscreen mode

The on mode behaved: the client said it would not take gzip, the module backed off, the original went out. always sent raw gzip bytes starting with 1f8b — to a client that had explicitly said otherwise. The documentation gives two justifications for this mode: when there are no uncompressed files on disk anyway, or when ngx_http_gunzip_module is used. That second clause is the answer to my own question about who rescues the client that cannot decode gzip: that module decompresses the response on the server on its way to such a client. Turn always on without it, and the answer is you.

How this file shows up in production

The lab fixture came together by hand; the real question is how such a file comes to exist in an actual deployment. The most common route is copying build output into the docroot without --delete:

$ printf 'v1 kisa govde\n' > dist/sayfa.html
$ gzip -9 -nc dist/sayfa.html > dist/sayfa.html.gz
$ rsync -a dist/ pub/
# second round: the build no longer emits .gz, only .html
$ rm -rf dist && mkdir dist
$ printf 'v2 govde biraz daha uzun oldu\n' > dist/sayfa.html
$ rsync -a dist/ pub/
$ ls -l pub/ | awk 'NR>1 {print $5, $9}'
30 sayfa.html
34 sayfa.html.gz
$ cat pub/sayfa.html;  gzip -dc pub/sayfa.html.gz
v2 govde biraz daha uzun oldu
v1 kisa govde
Enter fullscreen mode Exit fullscreen mode

Nobody made a mistake: the build produced the right file, rsync copied the right file. The .gz that no longer has a counterpart in the source simply stayed where it was, and from that moment on it became the body nginx sent to gzip clients. With --delete added, the same round left only sayfa.html in pub2; the orphan concept disappeared.

I walked into my own trap while measuring, and noting it is a debt of honesty: on the first attempt both versions had 9-byte bodies and I had written them within the same second. rsync's quick check skipped the file entirely because size and timestamp matched, and for a moment I was close to concluding "not even the original gets updated" — a wrong finding. Once I gave the bodies different lengths and put 1.2 seconds between them, the measurement started measuring the mechanism I actually wanted. The measuring rig produces findings of its own; you have to make sure it isn't.

After you fix it, the job is not over

Say you spot the orphan and refresh the .gz. When does nginx see that? With open_file_cache on, not immediately: alongside file descriptors, that cache also holds sizes and modification times, and open_file_cache_valid decides when they get revalidated.

That mechanism got a full end-to-end treatment in a separate article three days ago: growing a file to 5,000 bytes while nginx sent 100 — the stale cached size goes into Content-Length, the body gets cut, and the status code is still 200. Both lessons from there apply here unchanged, except that this time the truncated body is a gzip stream: rewrite the file in place (same inode) and the client gets half a stream; replace it atomically with mv and the client gets the whole of the old version. This blog's deploy also mvs the entire dist directory into place, so it falls in the second group.

One detail from that measurement is worth repeating, because it runs against intuition: the clock starts not when you fix the file but when the cache entry was born, and each worker carries its own entry. So you cannot say "I fixed the .gz, it will clear in 30 seconds". What you can say is "it stays stale for at most valid, and I do not know which second it turns". inactive is in play too: on a rarely requested path the entry is dropped anyway, so the window may never fill up in practice.

What would I actually gain on my own site

Now to the real question: is the risk worth it? I pulled my own site's real assets uncompressed and put them on the scale.

File Raw gzip -1 gzip -9 Gain
Home page HTML* 122,826 35,129 29,724 5,405 bytes
BaseLayout CSS 114,436 25,417 20,237 5,180 bytes
ClientRouter JS 13,888 5,308 4,763 545 bytes
TermHome CSS 5,645 1,513 1,321 192 bytes

* The HTML row is there for comparison only; that body is produced by SSR, there is no file on disk, so it is not something gzip_static could ever win. The only real candidates are the assets under /_astro/.

The level in my config is 1. Precompression's appeal is that it makes level 9 free: on my largest CSS, 20,237 bytes instead of 25,417 — a gain worth about a fifth of the compressed size.

The CPU side went on the clock too: compressing that 114 KiB CSS 200 times spent 0.20 seconds of user time at level 1 and 0.45 seconds at level 9. Those numbers include the cost of starting the gzip process each time; I ran the same loop against a 1-byte file, measured that share as 0.11 seconds, and subtracted it. The values I derived from there: roughly 0.45 ms per compression at level 1 and 1.70 ms at level 9. What went on the clock is not nginx's in-process zlib call but gzip in a separate process, so treat these as an upper bound rather than a precise figure.

Then I looked at my own site's live headers, and the arithmetic ended there:

$ curl -sI --compressed https://mustafaerbay.com.tr/_astro/BaseLayout.CGi080bn.css \
  | grep -Ei 'cf-cache-status|cache-control|age:'
cache-control: max-age=31536000
cache-control: public, immutable
age: 1713
cf-cache-status: HIT
Enter fullscreen mode Exit fullscreen mode

Hashed assets are immutable for a year and HIT at Cloudflare. The moments when my origin recompresses that CSS amount to the edge cache warming up. The HTML side is SSR: the body is produced from content.db, and there is no file on disk to precompress — gzip_static can do nothing on that path. As in the measurement where I grew a file and watched how many bytes nginx sent, the gain here is decided by the real distribution of traffic, not by theory.

So my table reads like this: a few thousand bytes of gain, a CPU saving that rarely triggers, and in exchange the orphan-file risk plus the cache's staleness window. On top of that there is no switch for me to flip — I would first have to take /_astro/ off the proxy and serve it from disk, which means changing the shape of the deploy. The interesting part is that without measuring, I would have turned it on as a "well-known optimization".

A decision frame

The list I wrote for myself, on when I would turn gzip_static on:

  • Does the build itself produce the compressed file? A gzip pass run by hand is a pass that gets skipped in the first rush.
  • Is the copy into the docroot done with --delete, or can orphan .gz files survive it?
  • Is there a check after deploy that every X.gz still has its X? A three-line find does the job, and as long as nobody looks, no warning ever comes.
  • Are the mtimes of the .gz and the original kept in sync? Nginx does not verify it, but your own check can.
  • Do monitoring requests send Accept-Encoding: gzip? If not, monitoring never sees the body users get.
  • If open_file_cache is on, open_file_cache_valid is the worst-case staleness you are accepting — it does not tell you when the window ends after your fix. Use an atomic swap rather than an in-place write: at least the client gets the old body instead of half a body.
  • If there is a CDN in front, look for the gain there first: if the edge is already compressing, the level difference at the origin may never reach a user.

What is left of it

Precompression means putting derived data on disk. Disk has no idea what that data was derived from: the a.html.gz file does not know it is a.html's child, and nginx never asks. That is why every "compute the result once and store it" optimization brings a freshness debt along with it — and the one who has to pay that debt is standing in the deployment pipeline, not in the config file.

My favourite detail across the whole measurement was that 304. Nginx was sending the wrong body, yet it was perfectly consistent about its wrongness: the size, the date, the ETag, all of them matched the file it was sending, byte for byte. That is how systems usually break — not through inconsistency, but through flawless loyalty to the wrong thing.

Official Sources

Top comments (0)