I Grew a File to 5,000 Bytes, nginx Sent 100

Wandering through nginx configurations, I keep running into the same four lines.
Always the same values, always the same order:

open_file_cache          max=1000 inactive=20s;
open_file_cache_valid    30s;
open_file_cache_min_uses 2;
open_file_cache_errors   on;

Looking for why this quartet comes out identical everywhere, I found it: this
block is nginx's own documentation Example:, verbatim — same values, same
order. That's also why min_uses is 2 in every template; nobody picked that
number for their own traffic, we are all copying the same example.

The name is lovely too. "Open file cache" — it sounds like something every static
file server ought to turn on. I considered adding it to my own blog's nginx, and
before doing so I asked a question: what exactly does this buy me, and what am I
giving up in return? I read the documentation; it wasn't enough. I read the source
code, and something quite different came out of it. Then I ran it on my own
machine, and the numbers put a behavior in front of me that the docs never
mention: a file I had grown in place to 5,000 bytes was served by nginx for 28
seconds as 200 OK with Content-Length: 100. No error log, no warning. As far
as the browser was concerned, everything was fine.

This article is the anatomy of those 28 seconds. And at the end, the story of why
I did not write those four lines into my own blog.

The lab

I ran the measurements inside the linuxkit virtual machine of Docker Desktop on my
own Mac. There's no point in changing files on a production server and breaking
live traffic; here, the only thing I can break is my own container.

Docker Engine 27.4.0 (Docker Desktop) · Linux 6.10.14-linuxkit aarch64
14 vCPU · MemTotal 5,157,470,208 bytes · root filesystem: overlay
nginx:alpine → nginx/1.31.6

I opened two server blocks in a single nginx instance. Same process, same
kernel, same disk — the only difference being whether the cache is on. That way I
eliminated the question "am I measuring the cache, or the machine's mood at that
moment?" from the start:

# :8081 — open_file_cache off (nginx's default)
server {
    listen 8081;
    root /srv/a;
    open_file_cache off;
    access_log off;
}
# :8082 — cache enabled
server {
    listen 8082;
    root /srv/b;
    open_file_cache          max=1000 inactive=20s;
    open_file_cache_valid    30s;
    open_file_cache_min_uses 1;
    open_file_cache_errors   on;
    access_log off;
}

I deliberately set min_uses to 1; the value of 2 that circulates in templates
makes the first request of a measurement invisible, and I wanted to see the
mechanism itself.

First I checked the benefit

If I'm going to defend a setting, I should know its gain first. I dropped to a
single worker, fired 300 requests at the same 4,096-byte file, and listened to the
worker with strace -c.

Syscall	open_file_cache off	open_file_cache on
openat	300	0
fstat	300	0
close	599	300
sendfile	300	300

The gain is clear: three syscalls per request. openat and fstat disappear
entirely, and close drops from 599 to 300 — with the cache off every request
opens and closes a file descriptor, with it on there is nothing left to close and
only the connections' own closes remain. The three together come to
300 + 300 + 299 = 899 calls; divided by three hundred requests, 2.997. The reason
the close count is 599 rather than 600 is that one close fell outside the strace
window; what's missing is the observation, not the call.

It is very tempting to stop here and say "so it's three times faster." strace
reported times between 10 and 22 µs for these calls. But I don't trust those
numbers, because strace slows down the very thing it measures: it stops and
inspects every syscall through ptrace. The usecs/call figures I have aren't the
real cost, they're an upper bound inflated by the observer's own overhead — which
is why I derive no timing claim from them.

So I removed the observer and measured end to end. ab with 20,000 requests,
keepalive on, concurrency 1. Three rounds with off first, three rounds with on
first — to eliminate ordering bias as well:

rounds with off first : off avg 16,711 rps | on avg 17,635 rps  (+5.53%)
rounds with on first  : off avg 17,486 rps | on avg 17,914 rps  (+2.45%)
────────────────────────────────────────────────────────────────────────
all 12 runs           : off avg 17,098 rps | on avg 17,775 rps  (+3.95%)
off range: 16,441 – 17,932 rps   (spread 1,491)
on  range: 17,194 – 18,577 rps   (spread 1,383)

The averages favor on, and so do five of the six pairings — all but one, where
in L8b's second round off at 17,497 beats on's 17,194. And looking a little
closer at the numbers: off's best run (17,932) is faster than on's worst run
(17,194). The two groups' ranges overlap. The difference between the averages is
676 rps, whereas the spread within off alone is 1,491 rps — the noise is
2.2 times the effect I'm trying to measure.

The conclusion I have to accept: in this setup I could not measure
open_file_cache's end-to-end gain. The syscall count is certain; its effect on
time stayed below my machine's own fluctuation. Three syscalls per request is a
real saving, but for it to mean anything the rate of file opens has to be very
high — how high, I did not measure, so I'm not going to invent a threshold
number. The one certain thing: on a blog that sees a few hundred static file
requests a day, you can't even measure what you gained.

So is the cost unmeasurable too? That part turned out quite differently.

I asked for 5,000 bytes and got 100

The scenario is as ordinary as I could make it: there's a 100-byte file, nginx
served it once, and two seconds later I grew the file in place to 5,000 bytes.
In place meaning with the shell's > redirection — that is, with the thing most
scripts, most editors, and most "update this file" commands do.

First I proved that this really is an in-place write, because the whole story
hangs on it. I ran the check on a separate /tmp file so as not to disturb the
one being served — the two numbers below belong to that file:

# the > redirection
inode before = 34071674     inode after = 34071674     → SAME
# replacement via mv
inode before = 34071674     inode after = 34071675     → CHANGED

The file is now 5,000 bytes on disk, its mtime updated, its inode unchanged
(34071666). I sent one request per second to both ports and recorded what came
back:

                 open_file_cache off            open_file_cache on
t=+0s    200  cl=5000  received=5000  first=B   200  cl=100  received=100  first=B
t=+1s    200  cl=5000  received=5000  first=B   200  cl=100  received=100  first=B
  ...                 (28 consecutive samples)                ...
t=+27s   200  cl=5000  received=5000  first=B   200  cl=100  received=100  first=B
t=+28s   200  cl=5000  received=5000  first=B   200  cl=5000 received=5000 first=B

The left side is what I expected: with the cache off, the file changed and nginx
gave its new state. The right side did this for 28 seconds — first byte B, so
the new content; length 100, so the old size. nginx made no new open
call — the descriptor it already held still pointed at the same inode, so it
could read the new content; it sent the first 100 bytes from that and threw away
4,900. This is also why openat came out at zero in the syscall count. And since Content-Length is also
100, there is no inconsistency from the client's point of view: 100 bytes were
promised, 100 bytes arrived, 200 OK. Nothing says that 98% of the content is
missing.

I checked the error log. Not a single line from start to finish of this experiment — no [error], no
[warn], no [alert]. nginx isn't even aware it did this, because from its own
point of view there is no error: it was asked to send 100 bytes, and it sent 100
bytes.

The timing isn't a coincidence either. The request that populated the cache went
out 2 seconds before I grew the file. open_file_cache_valid is 30 seconds.
30 − 2 = 28. I saw the transition at exactly t=+28s. The arithmetic lines up
precisely, which tells me what I captured is mechanism, not noise.

The source code says what it caches

I went back to the documentation. nginx.org lists what open_file_cache stores:
open file descriptors, their sizes and modification times; information on the
existence of directories; and file lookup errors. For open_file_cache_valid
there is a single sentence: it sets a time after which elements "should be
validated."

Validated how? The docs don't say. What gets compared — mtime, size? That gap is
exactly where my 28 seconds live, so I opened ngx_open_file_cache.c. Here is the
body of the cache hit:

if (file->use_event
    || (file->event == NULL
        && (of->uniq == 0 || of->uniq == file->uniq)
        && now - file->created < of->valid
        /* ... symlink checks omitted ... */
   ))
{
    if (file->err == 0) {
        of->fd    = file->fd;
        of->uniq  = file->uniq;
        of->mtime = file->mtime;
        of->size  = file->size;

I shortened one condition in that quote: the leading file->use_event branch
skips the valid check entirely and trusts file events coming from the kernel.
On Linux that branch never runs (vnode events belong to kqueue systems), which is
why my measurements are clean — but I should say that every result in this
article holds for Linux/epoll.

There's the answer. As long as now - file->created < of->valid holds, there is
no syscall at all here. The file descriptor comes from the cache — but so do
size and mtime. nginx writes the response's Content-Length not by looking at
the file on disk, but by recalling the size at the moment it created the entry.

A descriptor open on the same inode reads the new content — the file changed, but
that descriptor still points at the same file. So nginx takes the body from the
new content and the length from the old bookkeeping. When the two conflict, the
bookkeeping wins.

So where exactly does that cached size enter the response? In the static file
module, a single number determines two separate places at once:

/* ngx_http_static_module.c — the same of.size goes to two places */
r->headers_out.content_length_n   = of.size;
r->headers_out.last_modified_time = of.mtime;
/* ... */
b->file_pos  = 0;
b->file_last = of.size;

of.size writes the Content-Length header, and it also draws the b->file_last
boundary that tells sendfile how many bytes to send. The promise made to the
client and the number of bytes actually sent are fed from the same source; when
that source goes stale, both go stale together. That's why the response looks
internally consistent while being incomplete relative to the file. In the same
block last_modified_time comes from of.mtime too, so the Last-Modified
header also stays old throughout the window. More than that, nginx builds the
ETag out of those same two fields:

/* ngx_http_core_module.c — ngx_http_set_etag */
ngx_sprintf(etag->value.data, "\"%xT-%xO\"",
            r->headers_out.last_modified_time,
            r->headers_out.content_length_n);

The ETag is the hexadecimal pairing of a stale mtime with a stale size. So
that truncated 100-byte response goes out carrying an old Last-Modified and an
old ETag; a client or an intermediate cache arriving with If-None-Match or
If-Modified-Since is told "unchanged." That's why nothing looks inconsistent on
the client side: the response's entire bookkeeping agrees with itself, and
disagrees only with the file.

When I first read this, I thought: the benefit was saving three syscalls. The cost
is that same saving. The fstat I gave up was precisely the call that would have
told me how much the file had grown — by skipping it, the cache skips the time
and the truth together.

nginx doesn't look at mtime, it looks at the inode

What happens when the valid period expires? The code re-examines the path while
still holding the cached descriptor:

of->fd   = file->fd;
of->uniq = file->uniq;

rc = ngx_open_and_stat_file(name, of, pool->log);

And inside ngx_open_and_stat_file, the place where the decision is made:

if (of->fd != NGX_INVALID_FILE) {
    if (ngx_file_info_wrapper(name, of, &fi, log) == NGX_FILE_ERROR) {
        of->fd = NGX_INVALID_FILE;
        return NGX_ERROR;
    }
    if (of->uniq == ngx_file_uniq(&fi)) {
        goto done;
    }
}

What is ngx_file_uniq? src/os/unix/ngx_files.h says it in one line:

#define ngx_file_uniq(sb)        (sb)->st_ino

The inode. On revalidation nginx does not compare mtime, does not compare
size; it looks only at the inode number. mtime and size are copied from the
fresh stat result at the done: label, but they never enter the decision.

This produces a counterintuitive result. When you change a file in place, the
inode stays fixed, which means nginx's validation criterion never budges. When you
replace a file with mv, the inode changes and nginx sees the difference at once.

The measurement confirms it. I wrote 900 bytes into a temporary file and moved it
over a 300-byte file with mv — the classic atomic replacement. The inode went
from 34071676 to 34071674. On the cache-enabled side, the result:

t=+0s  ... t=+27s   200  cl=300  received=300  first=E   ← old file, complete
t=+28s             200  cl=900  received=900  first=F   ← new file

28 seconds again, but this time a consistent staleness. The body is the whole
of the old file, the length is the old file's length. A stale response, not a
broken one.

I rather enjoy the irony here: the move that looks "safe," updating a file in
place, produces a broken response. The move that looks "aggressive," deleting a
file and moving a new one into its place, produces only staleness. We've been
using atomic rename for years so that half-written files never become visible; it
turns out it has a second benefit in front of an nginx with open_file_cache
enabled.

Shrinking is loud, growing is quiet

I set up the same experiment in reverse: I shrank the 5,000-byte file in place to
100 bytes. The size in the cache is now larger than the file on disk.

t=+0s ... t=+27s   200  cl=5000  received=100  curl_rc=18
t=+28s             200  cl=100   received=100  curl_rc=0

(The loop skipped a second, so the 28-second window holds 27 samples rather than
28; the alert count below matches it exactly.)

curl exits with 18 — the response was cut short. nginx promised 5,000 bytes,
could find 100, and closed the connection. And this time it wrote to the error
log:

[alert] 87#87: *86 sendfile() reported that "/srv/b/shrink.html" was
truncated at 100 while sending response to client, client: 127.0.0.1,
server: _, request: "GET /shrink.html HTTP/1.1", host: "127.0.0.1:8082"

I counted this line 27 times within the 28-second window; in the growth scenario
there is no counterpart to it at all. The logic makes sense: when shrinking, the
kernel tells nginx "there aren't as many bytes as you asked for," and nginx
complains. When growing, the 100 bytes it asked for are more than available, and
nobody objects.

The upshot is that open_file_cache has two failure modes, and the dangerous one is
the quiet one. If your file shrinks, you get an alert-level log entry, the client
sees an error, and someone notices. If your file grows, nothing happens — the
response looks valid, it just has no ending. Picture the first 100 bytes of an
HTML file: a page that starts with <!doctype html> and stops in the middle.
200 OK.

A file that didn't exist went on existing for 28 seconds

I wanted to measure the open_file_cache_errors on line too, because it always
comes bundled in the templates. I requested a nonexistent path, got a 404, then
created the file:

file created: 200 bytes
                 off                        on
t=+0s    200  cl=200  received=200      404  cl=153  received=153
  ...          (28 consecutive samples)
t=+27s   200  cl=200  received=200      404  cl=153  received=153
t=+28s   200  cl=200  received=200      200  cl=200  received=200

The file is on disk, 200 bytes, readable. nginx said it wasn't there for 28
seconds.

Thinking "maybe it keeps trying but can't get permission," I listened to the
worker with strace: I sent 20 requests to the existing file, all returned 404, and
the number of openat, newfstatat, statx and fstat calls touching that path
came out zero. The decision is made entirely from the cached error record; the
disk is never consulted.

One more detail, the kind that will confuse you in operations: nginx writes a line
to the error log for these cached 404s as well —

[error] 87#87: *238 open() "/srv/b/sonra.html" failed
(2: No such file or directory)

The open() call was never made, but the record says open() failed, because the
error text is produced from the cached state. So while the log says "no such
file," the file is sitting right there when you look with ls. You don't want to
be untangling that pair at midnight.

The clock starts at first open, not last access

The last question I had left was this: if a file is requested continuously, does
the valid window get refreshed? If it does, a heavily trafficked file might
never update; if it doesn't, the window is a fixed deadline from the first open.

I ran it with two files. I first requested X at t=+0s and Y at t=+10s; at t=+14s
I grew both in place to 900 bytes — same instant, same change, same worker. From
t=+14s onward I requested both of them every second, so throughout the
measurement their traffic was identical. The only difference between them was the
moment their entries were born.

t=+29s   X=300B   Y=300B
t=+30s   X=900B   Y=300B     ← X refreshed
t=+39s   X=900B   Y=300B
t=+40s   X=900B   Y=900B     ← Y refreshed

X refreshed exactly 30 seconds after its first open, and Y exactly 30 seconds
after its own first open. The 10-second gap between them is identical to the gap
between the two entries' first-open times. Both were receiving uninterrupted
requests while this happened — so traffic simply doesn't concern the created
counter.

inactive enters here too, and most write-ups skip it. The block actually
carries two separate clocks: valid is an expiry counted from the entry's birth,
while inactive is an abandonment timer counted from its last access. An entry
nobody requests for the template's 20 seconds falls out of the cache, and the
next request does a fresh fstat. So the real staleness window is the smaller of
the two: on a busy path valid decides, while on a rarely requested path the
entry drops out via inactive before the window ever fills. In my measurements I
sent at least one request per second, which is why valid always decided.

The practical consequence matters: X stayed stale for 16 seconds, Y for 26. The
same file change, the same configuration, two different staleness durations.
valid is not a "freshness interval" but an expiry date counted from each entry's
own birth. So the sentence "it stays stale for at most 30 seconds" is true, but
"it fixes itself after 30 seconds" is not — which second it fixes itself depends on
when that entry was born, and you don't know when that was.

Fourteen workers, fourteen separate expiry dates

Running with worker_processes auto, my VM starts 14 workers. After requesting the
file for a while, I scanned every worker's /proc/<pid>/fd directory:

workers holding an fd for flap.html:
4029:1 4030:1 4031:1 4032:1 4033:1 4034:1 4035:1 4036:1
4037:1 4038:1 4039:1 4040:1 4041:1 4042:1
workers holding an fd = 14   total fds = 14

A single file, fourteen separate open file descriptors. Because open_file_cache is
not a shared zone; every worker has its own private cache. The documentation
doesn't say this outright, but the fd table does.

Combining that with the previous section gives the following — and I did not
measure this, I derived it from the two measurements: since each worker's entry is
born at that worker's own first open, a single file change produces as many
different expiry dates as there are workers. In theory, then, some requests could
get the new content while others get the old, and what the client sees could jump
back and forth across consecutive requests.

Here's where I have to be honest: I did not see that jumping in my own run.
After the rename I sampled at 12 requests per second for 50 seconds; up to t=+13s
every response was old (300 bytes), and from t=+14s onward every response was new
(900 bytes). Not a single mixed second. In my priming traffic the 14 workers must
have created their entries at moments very close to one another, so their windows
nearly coincided. The mechanism allows a mixed window; I couldn't trigger one.

The second line item of the cost: open file descriptors

With min_uses 1 I requested 50 separate files once each and counted the worker's
fd table:

fds held for the cache-ON root (/srv/b) : 50
fds held for the cache-OFF root (/srv/a):  0
the worker's total fd count             : 64

The cache really does what its name says: it keeps files open. That max=1000 in
the template deserves attention — but not in the way it looks. The documentation
says max sets the maximum number of elements in the cache, not of open
descriptors; entries that never pass the min_uses threshold, and cached error
records, hold no descriptor at all yet still occupy that thousand-seat quota. So
1,000 is a ceiling on descriptors rather than an equality: at most 1,000 open
files per worker, at most 14,000 across 14 workers. Even as a ceiling, it is a
number that intersects directly with worker_rlimit_nofile and system limits; I previously
measured who sets file descriptor limits on fs.file-max
and saw that in containerized environments these numbers come from unexpected
places. When writing max into a configuration, it's worth remembering that this
is not a memory setting but a file descriptor budget.

It's also worth touching on what min_uses actually does, because its name
misleads. According to the source code, if an entry doesn't reach the min_uses
threshold its descriptor is closed, but the entry itself stays in the cache; on
the next access the file is reopened. So min_uses 2 means "don't hold a
descriptor for one-off files" — it does not mean closing the staleness window.

A decision framework

After the measurements, these are the rules I set for myself:

• Don't enable it if the content of the files you serve can change at the same path. A script that writes in place, a directory synced with rsync, a file produced by a CMS — all of them are candidates for the growth scenario, which is to say candidates for silent truncation.
• If you do enable it, only ever replace files with an atomic rename. That downgrades a broken response to a stale response. A free improvement, and a good habit anyway.
• Paths whose contents never change are the best candidates. Content-hashed asset files (app.a1b2c3.js) by definition never change; there is no staleness there, only gain.
• Think about errors on separately. Its gain is real during 404 storms, but its cost is a support ticket of the form "the file I just uploaded isn't showing." It should enter the configuration as a decision independent of valid.
• Choose valid according to your deploy frequency. Thirty seconds is far too long on a system that deploys every minute; on one that deploys hourly it's as harmless as the gain is small.
• If you use try_files, proxying won't save you. I had assumed "a request that goes upstream opens no file"; the source code corrected me. ngx_http_try_files_module.c probes paths by calling ngx_open_cached_file with the location's own open_file_cache settings. It sets of.test_only = 1, but ngx_open_cached_file honors that flag only when there is no cache; with a cache configured the file really is opened and cached. So in the most common pattern of all — location / { try_files $uri @backend; } — the cache is live, and together with errors on a static file you just uploaded will keep falling through to the proxy for as long as valid. The only place the cache truly doesn't reach is a plain proxy_pass block that fetches the response body from upstream.
• If you get stuck, the exit is nginx -s reload. Because the cache is per-worker, every entry is wiped the moment a reload spawns new workers; that's the one-command way out of a stale window without waiting for valid.
• Set up something that can detect it. Growth is quiet, so it won't announce itself: a small post-deploy check comparing the Content-Length from curl -sI against stat -c %s on disk catches this article's whole scenario. For the shrinking side, grepping the error log for truncated at is enough.

What I did on my own blog

The last few points led me to my own configuration, and there I ran into an
unpleasant surprise: the first file I opened was the wrong one. The repository
holds deploy/nginx-mustafaerbay-final.conf, and the name sounds self-assured
enough; but deploy/nginx-live/README.md says that file is the clean-install
template, and that the copy actually running on the server is
deploy/nginx-live/sites-enabled_mustafaerbay.conf. I came close to ending this
article with a decision made from the wrong file.

Here is what the live file looks like: seventeen location blocks, seven of them
going through proxy_pass to the Astro SSR process at 127.0.0.1:3040, and a
group of them closing off scanner paths outright with return 444.
open_file_cache appears nowhere, so the default is off. Up to this point my
guess held: on my site nginx mostly doesn't serve files, it relays a
conversation.

But there are three places that read a file from disk:

location /.well-known/acme-challenge/ { root /var/www/letsencrypt; }

location /_admin/analytics/ {
    alias /var/www/analytics/;
    index mustafaerbay.html;
    auth_basic "Analytics";
    auth_basic_user_file /etc/nginx/.htpasswd-analytics;
    try_files $uri $uri/ =404;
}

location @astro_prev {
    root /opt/mustafaerbay/dist-old/client;
    try_files $uri =404;
    expires 1y;
    add_header Cache-Control "public, immutable" always;
}

The first is certbot's validation directory; a handful of files get read for a
few seconds while a certificate is renewed, and there's nothing for a cache to
win there.

The last is the fallback block that kicks in when an asset under /_astro/
returns 404 at the origin. There's no staleness risk there, because the files are
content-hashed and immutable: the same name always means the same content.
There's no gain either; that block only runs in a narrow window after a deploy,
while old HTML asks for a new asset, and it has expires 1y and Cloudflare in
front of it.

What actually stopped me was the one in the middle. My ops panel's HTML sits on
disk under a fixed name — no content hash, no immutable, and unlike the assets,
the content at that same address changes over time. Had I enabled the cache,
which of this article's two failure modes I'd meet would depend entirely on how
that file gets refreshed: written in place means a truncated panel, replaced by a
move means a stale one. I can't tell which from the repository, because the file
lives only on the server. I'd rather not tie a setting to the update habits of a
file whose behavior I can't see. On top of that, both of these blocks use
try_files — and as I just learned, the cache reaches in there too.

My deploy script also works by moving directories:

rm -rf "$DEST/dist-old"
[ -d "$DEST/dist" ] && mv "$DEST/dist" "$DEST/dist-old"
mv "$DEST/dist-new" "$DEST/dist"

dist-old becomes an entirely different directory on every deploy. The key in
the cache, meanwhile, is the full file path — the path stays the same while the
file behind it changes. Exactly the rename scenario from the lab: a consistent
but old response, for as long as valid.

My decision: I didn't enable it. The gain is three syscalls per request and I
can't even measure those three on my own traffic; in exchange the configuration
would acquire a trap contingent on how my panel gets updated, plus four lines
someone could later copy into the wrong place. I had already learned how easily
you can lose track of what actually gets read in an nginx configuration
the day I left sixty-eight backup files behind.
Today I refreshed that lesson by mixing up which file was the live one.

Closing

While working through this setting, one question got clearer in my head: when we
evaluate a cache, we usually ask "how much does it speed things up?" The better
question is this one — what truth does this cache forget, and for how long does it
forget it?

The truth open_file_cache forgets is the file's size. The duration of the
forgetting is valid. Once you learn both at once, the setting's name suddenly
reads more honestly: this isn't an accelerator, it's an agreement about how long
to remember. nginx is telling you, "I won't ask again about this file for 30
seconds." And in return you're accepting this: if the truth about that file
changes within those 30 seconds, I will tell the client what I remember rather
than what is true.

The sentence in the documentation that says it "can store open file descriptors,
their sizes and modification times" was stating the whole agreement from the
start. If it stores the size, it will say the wrong thing when the size changes. I
didn't understand that by reading the sentence; I understood it by going looking
for where 4,900 bytes had gone.

Official Sources

• nginx — ngx_http_core_module: open_file_cache
• nginx source — src/core/ngx_open_file_cache.c
• nginx source — src/os/unix/ngx_files.h (ngx_file_uniq)
• nginx — ngx_core_module: worker_processes
• nginx source — src/http/modules/ngx_http_try_files_module.c
• nginx source — src/http/ngx_http_core_module.c (ngx_http_set_etag)
• nginx source — src/http/modules/ngx_http_static_module.c

Comments

[Nnamdi Felix Ibe]

Mustafa, this is the best thing I've read in weeks, and I say that as someone who reached the strace table, thought "three syscalls, nice", and then had the rest of the article happen to him. 😄

The opening finding deserves a poster. Those four lines are nginx's own documentation Example copied verbatim, so min_uses 2 isn't a tuning decision anybody made. It's one anonymous decision inherited by everybody. You handed me the phrase "borrowed decisions" on Day 57, and I've used it ever since. This is the purest example of it I've seen.

What I recognised:

The silent one. On Day 55 I found that mounting /var/log/nginx over the official image hides the symlinks to stdout and stderr, so nginx starts writing real files nobody reads. Same shape as your growth scenario. nginx carries on, the response looks fine, the log says nothing, and the thing you gave up is the thing that would have told you. Shrinking gets an alert, growing gets silence. The failure mode that announces itself is the kind one.

The wrong file. You nearly closed the article on a decision made from nginx-mustafaerbay-final.conf while the server runs something else. On Day 60 I mounted a volume at /usr/share/nginx/html/data on an httpd image, watched the persistence test pass, and only later worked out that Apache serves from /usr/local/apache2/htdocs. The test was honest. It was answering a different question. You caught yours before publishing, which is the harder version.

That valid counts from the entry's birth rather than last access, and traffic never touches it. Your X and Y experiment is the cleanest demonstration of something I'd have got wrong under oath. "Stale for at most 30 seconds" is true. "Fixes itself after 30 seconds" isn't, and you can't know which second because you don't know when the entry was born.

And the ETag construction, which I'd never looked at. Last-Modified and Content-Length both come from the stale pair, and the ETag is those two hashed together. So the truncated response isn't just internally consistent. It's consistently signed.

One thing I'd add, and it's the part that would worry me in production:

The 28 seconds belong to nginx. The consequence doesn't have to. A truncated response emitted inside that window leaves carrying a valid-looking ETag and Last-Modified, and any cache downstream that stores it keeps it for its own TTL, which has no relationship to valid. The origin heals on schedule. The edge goes on serving 100 bytes until its own clock says otherwise. So "at most 30 seconds" is true of the component and not necessarily of the system, which makes your post-deploy check worth aiming at the edge as well as the origin.

Also, the cached 404 writing open() failed for a call that was never made is the most hostile log line I've encountered. ls shows the file. The log insists it isn't there. At midnight. 😅

Outstanding work, my friend. 🙏

[Mustafa ERBAY]

Nnamdi, this is exactly why I enjoy our discussions! 😄

You’re absolutely right about the downstream cache. I measured 28 seconds of incorrect behavior at the origin, but that doesn’t mean the consequences end after 28 seconds. A CDN could turn a temporary inconsistency into a much longer problem.

That’s actually a great addition to the experiment. Testing only the origin tells us when nginx recovers, not necessarily when users stop seeing the wrong content.

And your point about testing the wrong file made me laugh. 😂 Sometimes our tests are perfectly correct; we’re just asking the wrong system the wrong question.

Thanks for reading it so carefully, my friend. You genuinely added something valuable to the discussion! 🙏