<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: jun liu</title>
    <description>The latest articles on DEV Community by jun liu (@jun_liu_ca219b73a8f1489b5).</description>
    <link>https://dev.to/jun_liu_ca219b73a8f1489b5</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4191583%2F9400c76e-8551-4e25-a0e7-05dfda5dc6ab.png</url>
      <title>DEV Community: jun liu</title>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jun_liu_ca219b73a8f1489b5"/>
    <language>en</language>
    <item>
      <title>Your Server Died at 3 AM: Reading the Linux OOM Killer's Report (and Stopping It)</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Wed, 11 Nov 2026 03:16:54 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/your-server-died-at-3-am-reading-the-linux-oom-killers-report-and-stopping-it-e7h</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/your-server-died-at-3-am-reading-the-linux-oom-killers-report-and-stopping-it-e7h</guid>
      <description>&lt;p&gt;It happens to every ops person eventually. The on-call page fires at 03:12: a customer-facing node has stopped answering, yet the process list looks normal and CPU is low. You SSH in, and the first line of &lt;code&gt;dmesg&lt;/code&gt; tells you what actually happened:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Out of memory: Killed process 4123 (java) total-vm:31457280kB, anon-rss:28901376kB, file-rss:1024kB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The kernel did not crash. It did not run out of disk. It ran out of memory, and it made a decision for you: it picked one process and killed it, hard. That decision is the &lt;strong&gt;OOM killer&lt;/strong&gt;, and the moment you learn to read its report, half of "mystery" 3 AM incidents stop being mysteries.&lt;/p&gt;

&lt;p&gt;This post walks through how the OOM killer chooses its victim, how to read the kernel log fields that matter, and the specific knobs (systemd &lt;code&gt;OOMScoreAdjust&lt;/code&gt;, cgroup &lt;code&gt;MemoryMax&lt;/code&gt;/&lt;code&gt;MemoryHigh&lt;/code&gt;, and a few &lt;code&gt;sysctl&lt;/code&gt; values) that let you control &lt;em&gt;who&lt;/em&gt; dies and &lt;em&gt;how fast the box degrades&lt;/em&gt;. The goal is not to "add more memory." The goal is to make the failure contained, predictable, and easy to explain in the postmortem.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the OOM killer actually does
&lt;/h2&gt;

&lt;p&gt;The kernel keeps a per-process accounting of memory. When an allocation cannot be satisfied and the reclaimer has genuinely given up, the kernel walks the candidate process tree, computes a "badness" score for each one, and kills the highest.&lt;/p&gt;

&lt;p&gt;The score is roughly proportional to &lt;strong&gt;resident anonymous memory (RSS)&lt;/strong&gt;, because that is the memory that, once the process dies, is actually returned to the system. Virtual memory (&lt;code&gt;total-vm&lt;/code&gt;) mostly does not count: it includes &lt;code&gt;mmap&lt;/code&gt;'d files and address-space reservations that may never have touched RAM. That distinction is the most common source of operator confusion, and it is why the OOM killer sometimes appears to have "picked the wrong process."&lt;/p&gt;

&lt;p&gt;There are two things worth internalizing:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The victim is rarely the culprit.&lt;/strong&gt; A long-running JVM with a 12 GB heap will almost always outscore a small, misbehaving cron job, even if the cron job is the one that just spiked. The killer rewards the process that is &lt;em&gt;currently resident&lt;/em&gt;, not the one that &lt;em&gt;allocated too much&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Just add swap" is not a fix.&lt;/strong&gt; Swap delays the moment the killer fires, but on a host that is already paging hard it makes everything slow while the OOM still happens. Swap is a cushion for temporary pressure, not a remedy for a leak or an undersized box.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Reading the kernel report
&lt;/h2&gt;

&lt;p&gt;The kernel always leaves a trail. Read both of these; on many systems one of them is empty when you need it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dmesg -T &lt;span class="o"&gt;|&lt;/span&gt; grep -iE &lt;span class="s1"&gt;'killed process|out of memory|oom'&lt;/span&gt;
journalctl -k --since &lt;span class="s2"&gt;"2 hours ago"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -i oom
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The line you are looking for looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Out of memory: Killed process 4123 (java) total-vm:31457280kB, anon-rss:28901376kB, file-rss:1024kB, shmem-rss:0kB, UID:0 pguid:0
oom_score_adj:0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now, the field-by-field breakdown that actually matters:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;What it is&lt;/th&gt;
&lt;th&gt;What to do with it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;total-vm&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Virtual memory size. Mostly meaningless for diagnosis (includes reservations and file-backed maps).&lt;/td&gt;
&lt;td&gt;Do not use it to judge "how big this process is."&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-rss&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Resident anonymous (heap/stack) memory. This is what the killer cares about.&lt;/td&gt;
&lt;td&gt;Compare against the process's normal baseline.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;file-rss&lt;/code&gt; / &lt;code&gt;shmem-rss&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;File-backed and shared-memory resident pages. Reclaimable, lower priority.&lt;/td&gt;
&lt;td&gt;Usually not the story in an app OOM.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;oom_score_adj&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The tuning knob, range &lt;code&gt;-1000&lt;/code&gt; to &lt;code&gt;+1000&lt;/code&gt;.&lt;/td&gt;
&lt;td&gt;Tells you whether anything had already nudged this process up or down.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The one question the report forces you to answer: &lt;strong&gt;was the killed process growing without bound, or did some &lt;em&gt;other&lt;/em&gt; process just allocate a huge amount at once and the killer picked the incumbent?&lt;/strong&gt; Compare the killed process's &lt;code&gt;anon-rss&lt;/code&gt; against its normal baseline. If &lt;code&gt;mysqld&lt;/code&gt; normally sits at 2 GB and the report shows it at 7.5 GB, MySQL is your problem. If MySQL was at its usual 2 GB and a Java worker was at 16 GB on a 16 GB box, the Java worker is the problem and MySQL was the victim.&lt;/p&gt;

&lt;p&gt;You can inspect the live score for a running process, which is useful for deciding who to protect &lt;em&gt;before&lt;/em&gt; the next event:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cat /proc/&lt;span class="nv"&gt;$(&lt;/span&gt;pidof myapp&lt;span class="o"&gt;)&lt;/span&gt;/oom_score      &lt;span class="c"&gt;# the computed score right now&lt;/span&gt;
cat /proc/&lt;span class="nv"&gt;$(&lt;/span&gt;pidof myapp&lt;span class="o"&gt;)&lt;/span&gt;/oom_score_adj  &lt;span class="c"&gt;# any manual adjustment&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One caution: once the process is dead, its PID may be reused by something else. Reading &lt;code&gt;/proc/&amp;lt;old-pid&amp;gt;&lt;/code&gt; after the event does not recover the victim's old score.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two knobs that actually save you
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. &lt;code&gt;OOMScoreAdjust&lt;/code&gt; via systemd
&lt;/h3&gt;

&lt;p&gt;If there is a service you genuinely cannot afford to lose (the auth service, the database, &lt;code&gt;sshd&lt;/code&gt; itself), tell the kernel that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/system/sshd.service.d/oom.conf
&lt;/span&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;OOMScoreAdjust&lt;/span&gt;&lt;span class="o"&gt;=-&lt;/span&gt;&lt;span class="mi"&gt;900&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Conversely, mark a known memory hog as "kill me first" with &lt;code&gt;OOMScoreAdjust=500&lt;/code&gt;, so the blast radius stays inside the process you can live without.&lt;/p&gt;

&lt;p&gt;Two rules I follow strictly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Never set &lt;code&gt;-1000&lt;/code&gt; on more than a handful of processes.&lt;/strong&gt; &lt;code&gt;-1000&lt;/code&gt; makes a process OOM-immune. If too many critical processes are all immune, the kernel has nothing left to kill and you get a full host panic instead of one contained service restart.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Set it on the unit, not ad hoc.&lt;/strong&gt; &lt;code&gt;echo -500 &amp;gt; /proc/$(pidof sshd)/oom_score_adj&lt;/code&gt; resets the moment the process restarts. The systemd drop-in survives reboot and restarts.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  2. &lt;code&gt;MemoryHigh&lt;/code&gt; / &lt;code&gt;MemoryMax&lt;/code&gt; — the real lever
&lt;/h3&gt;

&lt;p&gt;This is the single most effective control, and it is the one I reach for first. Instead of deciding who dies &lt;em&gt;after&lt;/em&gt; memory is gone, you put a ceiling on each service &lt;em&gt;before&lt;/em&gt; it can run the box over:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;MemoryHigh&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;3.5G&lt;/span&gt;   &lt;span class="c1"&gt;# soft ceiling: aggressive reclaim + throttling above this&lt;/span&gt;
&lt;span class="na"&gt;MemoryMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4G&lt;/span&gt;      &lt;span class="c1"&gt;# hard ceiling: cgroup OOM fires here, kills only this unit&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The split is important:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;MemoryHigh&lt;/code&gt; is a &lt;strong&gt;soft&lt;/strong&gt; threshold. Above it the kernel applies heavy reclaim pressure and throttles the cgroup. The service slows down instead of dying.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;MemoryMax&lt;/code&gt; is the &lt;strong&gt;hard&lt;/strong&gt; ceiling. Crossing it triggers an OOM &lt;em&gt;inside that cgroup only&lt;/em&gt;, so a runaway worker takes its own siblings down and the supervisor restarts it cleanly, while the rest of the host is untouched.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A half-killed container (or a half-killed service) is a worse failure mode than a clean full restart. With &lt;code&gt;MemoryMax&lt;/code&gt; in place, the box-wide OOM killer essentially stops firing, because each unit is now responsible for its own bad days.&lt;/p&gt;

&lt;h2&gt;
  
  
  The sysctl knobs, and why to be careful
&lt;/h2&gt;

&lt;p&gt;There are three &lt;code&gt;vm.*&lt;/code&gt; values you will see in every blog post about this, and two of them you should probably not touch.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;vm.overcommit_memory&lt;/code&gt; has three values:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;0 (default)&lt;/strong&gt; — heuristic overcommit. The kernel guesses whether an allocation is reasonable. Most workloads are fine here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;1&lt;/strong&gt; — always overcommit. &lt;code&gt;malloc&lt;/code&gt; essentially never returns &lt;code&gt;NULL&lt;/code&gt;. Useful for HPC workloads that allocate huge sparse arrays they never touch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;2&lt;/strong&gt; — strict accounting. The kernel refuses allocations that exceed &lt;code&gt;swap + RAM * overcommit_ratio / 100&lt;/code&gt;. This sounds appealing ("fail loudly instead of killing later"), but on a multi-tenant box it is painful: &lt;code&gt;fork()&lt;/code&gt; temporarily doubles the parent's virtual size, and many processes that normally work fine will fail to fork. JVMs in particular hate mode 2.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;code&gt;vm.panic_on_oom&lt;/code&gt; controls whether an OOM is upgraded to a full kernel panic instead of a kill. Leave it at &lt;code&gt;0&lt;/code&gt; unless you have a very specific reason; you want a contained service death, not a reboot.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;vm.swappiness&lt;/code&gt; is the one you &lt;em&gt;can&lt;/em&gt; tune, and it only moves the timing of pressure, not the outcome. A lower value (say &lt;code&gt;10&lt;/code&gt;) keeps file cache resident longer and pushes anonymous pages to swap sooner — useful on a cache-heavy box. It does not prevent an OOM; it only changes how the box degrades before one.&lt;/p&gt;

&lt;p&gt;The practical check before you walk into any of this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sysctl vm.panic_on_oom vm.overcommit_memory vm.overcommit_ratio vm.swappiness
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Container workloads
&lt;/h2&gt;

&lt;p&gt;For a containerized service the picture inverts, and you want the inversion: you &lt;em&gt;want&lt;/em&gt; the whole cgroup to die together rather than a single process inside it, because a half-alive container is not recoverable.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/systemd/system/worker.service.d/memory.conf&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;Service&lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="nv"&gt;MemoryMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;2G
&lt;span class="nv"&gt;MemoryHigh&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1.8G
&lt;span class="c"&gt;# cgroup v2: group OOM kills the whole cgroup, not one task&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On cgroup v2 with &lt;code&gt;memory.oom.group&lt;/code&gt; enabled, a limit breach kills the entire cgroup, which is exactly the clean-restart behavior you want for a worker fleet. On cgroup v1 the per-cgroup OOM still targets one process; the group behavior is newer. Check which one you are on with &lt;code&gt;stat -fc %T /sys/fs/cgroup&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  OOM is a symptom, not the cause
&lt;/h2&gt;

&lt;p&gt;Everything above decides &lt;em&gt;who&lt;/em&gt; dies. It does nothing about &lt;em&gt;why&lt;/em&gt; memory ran out. The fix is the cause:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A memory leak (RSS grows monotonically under steady load) → profile the process; in a JVM, take a heap dump after the event; in Go, grab a heap profile from pprof and watch what is retaining memory.&lt;/li&gt;
&lt;li&gt;An unbounded cache → set a real bound (L1 cache size, or cap the query result set).&lt;/li&gt;
&lt;li&gt;A query that loads a whole table into memory → limit it, stream it, or add an index.&lt;/li&gt;
&lt;li&gt;A host that is simply undersized for the workload → the honest answer is more RAM, or shed the workload.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Sizing the application and capping the cgroup are the remediation; the score adjustments are only deciding the order of death. If OOM events have recurred more than once and the pattern is still unclear, that is a monitoring-and-support investigation, and the running-host checklist for reading exactly these kernel fields lives in the ops notes on memory monitoring (deep link at the end of this post).&lt;/p&gt;

&lt;h2&gt;
  
  
  The one-paragraph summary for your postmortem
&lt;/h2&gt;

&lt;p&gt;The box ran out of memory and the kernel's OOM killer chose a victim by highest resident-anonymous score. I read the &lt;code&gt;dmesg&lt;/code&gt; report, confirmed whether the killed process was the culprit or an innocent incumbent by comparing its &lt;code&gt;anon-rss&lt;/code&gt; to baseline, then applied two controls: &lt;code&gt;MemoryHigh&lt;/code&gt;/&lt;code&gt;MemoryMax&lt;/code&gt; on the service cgroups so failures stay contained, and a negative &lt;code&gt;OOMScoreAdjust&lt;/code&gt; on the handful of processes that must survive. I did not add swap to "fix" it, because swap only delays and slows. The remaining work is the root cause — a leak, an unbounded cache, or an undersized host — which is the only thing that stops the next 3 AM page.&lt;/p&gt;

&lt;p&gt;These notes on memory pressure, cgroup limits, and postmortem playbooks are part of my wider field notes on running Linux in production. The running-host checklist that backs this post sits in the &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;travel-and-ops notes at jiunotes.top/en&lt;/a&gt;, and the memory-monitoring reference I kept referring to above is at &lt;a href="https://ops.jiunotes.top/en/monitor/linux-memory-monitor.md" rel="noopener noreferrer"&gt;ops.jiunotes.top/en/monitor/linux-memory-monitor.md&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>linux</category>
      <category>sre</category>
    </item>
    <item>
      <title>Self-Hosting Behind a Cloudflare Tunnel: Exposing Services Without Opening Inbound Ports</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Mon, 09 Nov 2026 23:03:57 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/self-hosting-behind-a-cloudflare-tunnel-exposing-services-without-opening-inbound-ports-4jo3</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/self-hosting-behind-a-cloudflare-tunnel-exposing-services-without-opening-inbound-ports-4jo3</guid>
      <description>&lt;p&gt;A Cloudflare Tunnel replaces inbound port forwarding with an outbound, encrypted connection: cloudflared on your machine dials out to Cloudflare's edge and holds it open. Visitors reach your hostname at Cloudflare, and traffic rides back over that existing connection. No public IP, no open ports, no router rules — just TLS, DDoS filtering, and optionally Access-based identity checks on top.&lt;/p&gt;

&lt;p&gt;I retired our last port-forwarding rule the day I stood up Cloudflare Tunnel on the homelab rack. We had a Home Assistant box, a small internal web dashboard, and a management SSH host — all reachable only because our ISP handed out a public IPv4 and we were brave enough to punch holes in the router. The moment that public IP churned during a firmware update, everything went dark for an hour, and every hour in between, our ports were open to the entire internet. After switching to a tunnel, the router rules died, the firewall became a straight deny-all, and access kept working through the churn. This post is the working record of how that setup actually behaves in production, not marketing copy.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the architecture inverts the connection
&lt;/h2&gt;

&lt;p&gt;With port forwarding, the internet initiates: your router advertises a listening port, scanners find it, and your service sits exposed behind a rule you must patch and babysit forever. A tunnel does the opposite. The &lt;code&gt;cloudflared&lt;/code&gt; daemon on your origin machine makes an outbound connection to Cloudflare's edge — over TLS, using QUIC by default — and holds it. Visitors hit your hostname, which resolves to Cloudflare, and Cloudflare hands the request back down the already-open connection to your origin.&lt;/p&gt;

&lt;p&gt;Two consequences matter operationally:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;No port on your network is ever inbound-reachable, so the attack surface shrinks to "the hostnames Cloudflare exposes for you."&lt;/li&gt;
&lt;li&gt;Your public IP — or CGNAT address, or the fact that you have no public IP at all — is irrelevant to reachability. That alone killed an entire class of outages for us.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The trade-off: your availability now depends on three things instead of one — your host is up, &lt;code&gt;cloudflared&lt;/code&gt; is running, and Cloudflare's network is healthy. Budget for that. Also understand that traffic transits Cloudflare, so it is no more end-to-end encrypted than any site behind the CDN; for privacy-sensitive services the mitigation is Access-restricting who can even reach it (see &lt;a href="https://ops.jiunotes.top/en/network/zero-trust-network" rel="noopener noreferrer"&gt;zero-trust networking&lt;/a&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  Install cloudflared
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;cloudflared&lt;/code&gt; is a single static binary from Cloudflare. On Debian/Ubuntu I use the official apt repo so it gets security updates with the normal &lt;code&gt;unattended-upgrades&lt;/code&gt; cycle:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg &lt;span class="o"&gt;|&lt;/span&gt; sudo tee /usr/share/keyrings/cloudflare-main.gpg &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null

&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared any main'&lt;/span&gt; \

  &lt;span class="o"&gt;|&lt;/span&gt; sudo tee /etc/apt/sources.list.d/cloudflared.list

sudo apt update &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; sudo apt install cloudflared

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The binary also ships an RPM repo and installers for macOS/Windows, and for containers a single &lt;code&gt;cloudflare/cloudflared&lt;/code&gt; image works. The full source lives at &lt;a href="https://github.com/cloudflare/cloudflared" rel="noopener noreferrer"&gt;https://github.com/cloudflare/cloudflared&lt;/a&gt; and the product docs at &lt;a href="https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/" rel="noopener noreferrer"&gt;https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/&lt;/a&gt; .&lt;/p&gt;

&lt;h2&gt;
  
  
  Two authentication modes: token vs credentials-file
&lt;/h2&gt;

&lt;p&gt;You have to tell the daemon which tunnel it belongs to. There are two ways, and I use both in different places:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Token mode.&lt;/strong&gt; In the Zero Trust dashboard you create a tunnel, copy a base64 token, and run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cloudflared tunnel run &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;your-tunnel-token&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the whole setup: no config file, no DNS routing (the dashboard creates the CNAME for you). It is perfect for a quick test, a temporary container, or a machine I do not want extra state on. The downside for long-lived infrastructure is that the secret lives in the process command line or a hand-maintained env file.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Credentials-file mode.&lt;/strong&gt; This is what I run everywhere permanent. One-time bootstrap:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cloudflared tunnel login

cloudflared tunnel create homelab

cloudflared tunnel route dns homelab ha.example.com

cloudflared tunnel route dns homelab dash.example.com

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;login&lt;/code&gt; opens a browser, you authorize the zone, and &lt;code&gt;create&lt;/code&gt; drops a tunnel UUID plus a JSON credentials file into &lt;code&gt;~/.cloudflared/&lt;/code&gt;. &lt;code&gt;route dns&lt;/code&gt; automatically creates the CNAME records so &lt;code&gt;ha.example.com&lt;/code&gt; and &lt;code&gt;dash.example.com&lt;/code&gt; point into the tunnel — no manual DNS, and it just works.&lt;/p&gt;

&lt;h2&gt;
  
  
  Writing the service config: /etc/cloudflared/config.yml
&lt;/h2&gt;

&lt;p&gt;When the daemon runs as a system service it reads &lt;code&gt;/etc/cloudflared/config.yml&lt;/code&gt;, not your home directory — that distinction has bitten me more than once. The file in production looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;tunnel&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;3f9c2ab1-6e7d-4a8c-9b0e-1d2e3f4a5b6c&lt;/span&gt;

&lt;span class="na"&gt;credentials-file&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/etc/cloudflared/3f9c2ab1-6e7d-4a8c-9b0e-1d2e3f4a5b6c.json&lt;/span&gt;

&lt;span class="na"&gt;ingress&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ha.example.com&lt;/span&gt;

    &lt;span class="na"&gt;service&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http://127.0.0.1:8123&lt;/span&gt;

  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;dash.example.com&lt;/span&gt;

    &lt;span class="na"&gt;service&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http://127.0.0.1:3000&lt;/span&gt;

    &lt;span class="na"&gt;originRequest&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

      &lt;span class="na"&gt;connectTimeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"10s"&lt;/span&gt;

      &lt;span class="na"&gt;responseHeaderTimeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"60s"&lt;/span&gt;

  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ops.example.com&lt;/span&gt;

    &lt;span class="na"&gt;service&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ssh://127.0.0.1:22&lt;/span&gt;

  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;service&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http_status:404&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notes from real use:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The final &lt;code&gt;service: http_status:404&lt;/code&gt; catch-all rule is mandatory; without it cloudflared refuses to start.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;ssh://&lt;/code&gt; entries are not HTTP — visitors get a browser-based web terminal, so I can reach the host even from a laptop with no client installed.&lt;/li&gt;
&lt;li&gt;Per-service &lt;code&gt;originRequest&lt;/code&gt; timeouts are where I tune upstream 502s (more below).&lt;/li&gt;
&lt;li&gt;Add &lt;code&gt;tunnel&lt;/code&gt; and &lt;code&gt;credentials-file&lt;/code&gt; at the top, keep &lt;code&gt;credentials-file&lt;/code&gt; owned by the service user, mode &lt;code&gt;0600&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Run it as a systemd unit
&lt;/h2&gt;

&lt;p&gt;I never leave a tunnel on a terminal. The unit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight systemd"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/system/cloudflared.service
&lt;/span&gt;
&lt;span class="k"&gt;[Unit]&lt;/span&gt;

&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cloudflared&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;managed&lt;/span&gt; &lt;span class="n"&gt;by&lt;/span&gt; &lt;span class="n"&gt;me&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="n"&gt;at&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;cloudflared&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;yml&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;network&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;online&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;

&lt;span class="na"&gt;Wants&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;network&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;online&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;

&lt;span class="na"&gt;ExecStart&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;cloudflared&lt;/span&gt; &lt;span class="n"&gt;service&lt;/span&gt; &lt;span class="n"&gt;run&lt;/span&gt;

&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;always&lt;/span&gt;

&lt;span class="na"&gt;RestartSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2s&lt;/span&gt;

&lt;span class="na"&gt;User&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cloudflared&lt;/span&gt;

&lt;span class="na"&gt;StateDirectory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cloudflared&lt;/span&gt;

&lt;span class="na"&gt;Group&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cloudflared&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;

&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;multi&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo useradd -r -s /usr/sbin/nologin cloudflared

sudo cp ~/.cloudflared/config.yml /etc/cloudflared/config.yml

sudo cp ~/.cloudflared/3f9c2ab1-*.json /etc/cloudflared/

sudo chown -R cloudflared:cloudflared /etc/cloudflared

sudo chmod 600 /etc/cloudflared/*.json

sudo systemctl &lt;span class="nb"&gt;enable&lt;/span&gt; --now cloudflared

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;Restart=always&lt;/code&gt; plus &lt;code&gt;RestartSec=2s&lt;/code&gt; is the safety net: the flaps I see in the logs self-heal within seconds. The critical rule I follow before every config change — run &lt;code&gt;cloudflared tunnel run homelab&lt;/code&gt; in the foreground against a staging copy first, watch it register, only then update the service file and &lt;code&gt;systemctl restart cloudflared&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Put Cloudflare Access in front of everything non-public
&lt;/h2&gt;

&lt;p&gt;A tunnel is a pipe; identity is the lock. For &lt;code&gt;ops.example.com&lt;/code&gt; (SSH) and the dashboard I register each hostname as a self-hosted application in the Zero Trust console and add one policy: allow my email domain, require one-time PIN or SSO. The effect: an attacker scanning the internet finds Cloudflare's edge IPs, not mine, and gets an identity challenge before the request ever touches the origin. This is the standard zero-trust pattern — validate identity at the edge, not the firewall — and I keep the full runbook in &lt;a href="https://ops.jiunotes.top/en/network/zero-trust-network" rel="noopener noreferrer"&gt;zero-trust network design&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;On SSL mode: for tunnel hostnames I use &lt;strong&gt;Full (Strict)&lt;/strong&gt; so Cloudflare validates the origin certificate end to end. For internal services with self-signed certs behind the tunnel, the simpler path is &lt;code&gt;noTLSVerify: true&lt;/code&gt; under &lt;code&gt;originRequest&lt;/code&gt; rather than weakening the edge mode. The Access and TLS settings both live under &lt;a href="https://developers.cloudflare.com/zero-trust/access/" rel="noopener noreferrer"&gt;https://developers.cloudflare.com/zero-trust/access/&lt;/a&gt; .&lt;/p&gt;

&lt;h2&gt;
  
  
  Troubleshooting: what I actually see
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;502 Bad Gateway.&lt;/strong&gt; The tunnel is fine; the origin is not. My first move is always &lt;code&gt;curl -v http://127.0.0.1:8123&lt;/code&gt; on the host itself. Second move: check I did not point the service at a stale port after a config migration. Third: tune &lt;code&gt;originRequest.responseHeaderTimeout&lt;/code&gt; — I had a dashboard whose cold start took 45s, and the default timeout produced exactly the "502 upstream timeout" symptom until I set it to 60s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tunnel flapping / reconnecting.&lt;/strong&gt; Read &lt;code&gt;journalctl -u cloudflared&lt;/code&gt; before touching anything — the log names the reason. Our flaps turned out to be (a) a cellular uplink with a 90s NAT idle timeout, and (b) &lt;code&gt;cloudflared&lt;/code&gt; being OOM-killed on a 1GB VPS during log bursts. In both cases the reconnect reason in the journal was unambiguous.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Certificate/credential errors.&lt;/strong&gt; If you moved the credentials file and forgot to chown it to the service user, the daemon fails to authenticate and logs a 401-style rejection. Same class of bug: JSON mode 600 owned by root, service running as &lt;code&gt;cloudflared&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DNS not resolving.&lt;/strong&gt; After &lt;code&gt;route dns&lt;/code&gt;, the CNAME propagates within minutes; &lt;code&gt;dig ha.example.com&lt;/code&gt; should show a target at &lt;code&gt;*.cfargotunnel.com&lt;/code&gt;. If it resolves to the wrong edge or stale IP, check that the zone's nameservers are actually Cloudflare.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Free-tier limits.&lt;/strong&gt; Cloudflare's terms prohibit using free tunnels as streaming/video servers (Plex, Jellyfin distribution), and outbound latency on free tunnels goes through their public return path — fine for management and dashboards, wrong tool for game servers or UDP-heavy workloads.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Operate it like infrastructure, not a toy
&lt;/h2&gt;

&lt;p&gt;Tunnels fail quietly: a deleted CNAME, a stale credentials file, an OOM-killed daemon, and the hostnames just stop working with no alarm at the origin. My minimum discipline: an external synthetic monitor probing &lt;code&gt;ha.example.com&lt;/code&gt; for 2xx from outside the network; &lt;code&gt;cloudflared&lt;/code&gt; alive-check plus edge latency in the same dashboard; and before any ingress change, a foreground test run plus a &lt;code&gt;dig&lt;/code&gt; to confirm the CNAME. When I needed to reach the homelab through the old NAT world for anything else, &lt;a href="https://ops.jiunotes.top/en/network/iptables-port-forwarding" rel="noopener noreferrer"&gt;port forwarding with iptables&lt;/a&gt; is the reference I keep handy — but I now use it only for the router's own management, never for apps. And because the tunnel terminates at Cloudflare while my internal zone still resolves through a local caching resolver, the whole stack sits on top of the &lt;a href="https://ops.jiunotes.top/en/network/dnsmasq-config" rel="noopener noreferrer"&gt;dnsmasq configuration&lt;/a&gt; that serves the internal names.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Why not just keep a VPS + WireGuard and skip the tunnel entirely?&lt;/strong&gt; For a handful of always-reachable internal services, a tunnel is dramatically less to operate: no VPN clients on visitor devices, no key distribution, no VPS to keep patching. I keep a WireGuard site-to-site link as a second path to the rack; the tunnel is for things that must be reachable by hostname from anywhere.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does the tunnel expose my whole internal network?&lt;/strong&gt; Only what you write in &lt;code&gt;ingress&lt;/code&gt;. Every rule names a specific hostname and a specific origin address, so the blast radius is exactly the services you listed. My red line: never a catch-all rule pointing at an internal subnet.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What about latency?&lt;/strong&gt; Expect one extra network hop through Cloudflare's edge. For dashboards and shell access it is noise; for latency-sensitive anything, measure first — that is the whole point of the free-tier caveat above.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related articles
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://ops.jiunotes.top/en/network/iptables-port-forwarding" rel="noopener noreferrer"&gt;iptables port forwarding: NAT rules and what to stop exposing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://ops.jiunotes.top/en/network/zero-trust-network" rel="noopener noreferrer"&gt;Zero-trust network design: from perimeter to identity&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://ops.jiunotes.top/en/network/dnsmasq-config" rel="noopener noreferrer"&gt;dnsmasq configuration: a local DNS cache for the internal zone&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;My wider collection of travel notes, guides and field tools lives at &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;jiunotes.top&lt;/a&gt;. The ops runbooks and troubleshooting notes that back this walkthrough are published on the same site at &lt;a href="https://ops.jiunotes.top" rel="noopener noreferrer"&gt;ops.jiunotes.top&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>infrastructure</category>
      <category>networking</category>
      <category>security</category>
    </item>
    <item>
      <title>When "Network Is Down" but the Server Is Fine: Debugging DNS Resolution Failures</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Sun, 08 Nov 2026 22:53:21 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/when-network-is-down-but-the-server-is-fine-debugging-dns-resolution-failures-53cf</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/when-network-is-down-but-the-server-is-fine-debugging-dns-resolution-failures-53cf</guid>
      <description>&lt;p&gt;You ping the server. It answers. You open the app in a browser. Connection refused — or worse, it hangs for thirty seconds and then times out. The server is up, the cable is plugged in, yet nothing resolves. Nine times out of ten, the problem is not the network at all. It is DNS.&lt;/p&gt;

&lt;p&gt;I have spent more than a few late nights on this exact symptom. Here is the field-tested path I now follow every time, so you do not have to reinvent it at 2 a.m.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with the cheapest test: &lt;code&gt;nslookup&lt;/code&gt; / &lt;code&gt;dig&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Before touching configs, confirm whether name resolution itself works:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig +short example.com
nslookup example.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Empty output or &lt;code&gt;SERVFAIL&lt;/code&gt; → the resolver is the problem, not the remote host.&lt;/li&gt;
&lt;li&gt;A correct IP but the app still fails → resolution is fine; look at the app or firewall next.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;NXDOMAIN&lt;/code&gt; → the name genuinely does not exist (typo, or the record was deleted).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On Linux, &lt;code&gt;systemd-resolved&lt;/code&gt; often listens on &lt;code&gt;127.0.0.53&lt;/code&gt;. If you point tools at the wrong resolver you get confusing results, so check &lt;code&gt;/etc/resolv.conf&lt;/code&gt; first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The usual suspects
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;/etc/resolv.conf&lt;/code&gt; got overwritten.&lt;/strong&gt; Cloud-init, DHCP leases, and VPN clients all rewrite this file. If &lt;code&gt;nameserver&lt;/code&gt; points at a dead IP, every lookup fails while pings to raw IPs still work.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A flaky internal resolver.&lt;/strong&gt; Pointing everything at one corporate DNS server that intermittently drops queries produces the classic "works, then doesn't" pattern.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;systemd-resolved&lt;/code&gt; stub confusion.&lt;/strong&gt; &lt;code&gt;127.0.0.53&lt;/code&gt; is the stub. Query it directly with &lt;code&gt;dig @127.0.0.53 example.com&lt;/code&gt; to isolate whether the stub or the upstream is at fault.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;nscd&lt;/code&gt; / &lt;code&gt;dnsmasq&lt;/code&gt; cache poisoning or staleness.&lt;/strong&gt; A cached &lt;code&gt;NXDOMAIN&lt;/code&gt; can linger. Flush with &lt;code&gt;sudo systemd-resolve --flush-caches&lt;/code&gt; (or restart &lt;code&gt;dnsmasq&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Firewall blocking UDP 53.&lt;/strong&gt; Outbound DNS is UDP/53 (and TCP/53 for large responses). A strict egress rule that allows TCP but not UDP will let &lt;code&gt;dig&lt;/code&gt; sometimes work and the app sometimes fail.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  A 5-minute triage checklist
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1. Does raw resolution work?&lt;/span&gt;
dig +short example.com

&lt;span class="c"&gt;# 2. Which resolver are we using?&lt;/span&gt;
&lt;span class="nb"&gt;cat&lt;/span&gt; /etc/resolv.conf

&lt;span class="c"&gt;# 3. Is the stub resolving, or just the upstream?&lt;/span&gt;
dig @127.0.0.53 example.com
dig @8.8.8.8 example.com

&lt;span class="c"&gt;# 4. Is UDP 53 actually open outbound?&lt;/span&gt;
nc &lt;span class="nt"&gt;-vzu&lt;/span&gt; 8.8.8.8 53

&lt;span class="c"&gt;# 5. Clear any poison cache&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemd-resolve &lt;span class="nt"&gt;--flush-caches&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If step 2 shows a dead nameserver, fix that first — it explains everything.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix that actually sticks
&lt;/h2&gt;

&lt;p&gt;For servers, I stopped relying on DHCP-assigned resolvers. I pin a known-good pair in &lt;code&gt;/etc/systemd/resolved.conf&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Resolve]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;DNS&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;8.8.8.8 1.1.1.1&lt;/span&gt;
&lt;span class="py"&gt;FallbackDNS&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;9.9.9.9&lt;/span&gt;
&lt;span class="py"&gt;Domains&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;~.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then &lt;code&gt;sudo systemctl restart systemd-resolved&lt;/code&gt;. The &lt;code&gt;Domains=~.&lt;/code&gt; line is the part people miss: it tells the resolver to use these servers for &lt;em&gt;every&lt;/em&gt; domain, not just ones matching a search list.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this matters beyond the outage
&lt;/h2&gt;

&lt;p&gt;DNS failures are silent and asymmetric. The database host resolves, the cache host does not, and suddenly one service throws timeouts while dashboards stay green. Baking a deterministic resolver config into the base image — and alerting on &lt;code&gt;dig&lt;/code&gt; latency, not just on ping — has saved me more 2 a.m. pages than any other single change.&lt;/p&gt;

&lt;p&gt;If you want the full postmortem write-up with packet captures and the exact &lt;code&gt;tcpdump&lt;/code&gt; filter I use to catch dropped UDP/53 queries, it lives on my ops notes site: &lt;a href="https://ops.jiunotes.top/en" rel="noopener noreferrer"&gt;ops.jiunotes.top/en&lt;/a&gt;. The same notes are mirrored on the main site: &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;jiunotes.top/en&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>cli</category>
      <category>debugging</category>
      <category>devops</category>
      <category>networking</category>
    </item>
    <item>
      <title>Inode Exhaustion: The Disk Full Error That Makes No Sense</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Sun, 08 Nov 2026 03:32:03 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/inode-exhaustion-the-disk-full-error-that-makes-no-sense-mof</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/inode-exhaustion-the-disk-full-error-that-makes-no-sense-mof</guid>
      <description>&lt;p&gt;You run &lt;code&gt;df -h&lt;/code&gt; and the root partition shows 85% used. Plenty of space left, right? Then you try to create a new file and get: &lt;code&gt;No space left on device&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;You check again. You reboot. You even resize the partition. Same error.&lt;/p&gt;

&lt;p&gt;The problem isn't your disk space. It's your inodes.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Are Inodes?
&lt;/h2&gt;

&lt;p&gt;An inode is a data structure that stores metadata about a file: permissions, owner, timestamps, and pointers to the actual data blocks on disk. Every file, every directory, every symlink gets one inode.&lt;/p&gt;

&lt;p&gt;When you create a file, the system needs two things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Data blocks&lt;/strong&gt; to store the actual content (governed by disk space)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An inode&lt;/strong&gt; to describe the file (governed by inode count)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are separate limits. &lt;code&gt;df -h&lt;/code&gt; only shows data block usage. It says nothing about inodes.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Classic Scenario
&lt;/h2&gt;

&lt;p&gt;Here's a real example I saw last month on a production server hosting a Node.js application:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;df&lt;/span&gt; &lt;span class="nt"&gt;-h&lt;/span&gt;
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   85G   15G  85% /

&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;df&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt;
Filesystem     Inodes IUsed   IFree IUse% Mounted on
/dev/sda1     6553600 6553600       0  100% /
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The disk is 85% full but has zero inodes remaining. That's the contradiction that confused the on-call engineer for an hour.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Causes Inode Exhaustion?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Tiny Files Accumulate
&lt;/h3&gt;

&lt;p&gt;Every npm install, every webpack cache entry, every log rotation creates thousands of small files. A single &lt;code&gt;node_modules&lt;/code&gt; directory can easily contain 50,000+ files.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Count files in node_modules&lt;/span&gt;
find &lt;span class="nb"&gt;.&lt;/span&gt; &lt;span class="nt"&gt;-type&lt;/span&gt; f | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;
&lt;span class="c"&gt;# Often: 80000-150000 files&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Log Rotation Gone Wrong
&lt;/h3&gt;

&lt;p&gt;Misconfigured logrotate creates compressed copies without deleting originals. Or a backup script writes hourly snapshots to &lt;code&gt;/tmp&lt;/code&gt; without cleanup:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Culprit pattern: backup creates a file per minute&lt;/span&gt;
&lt;span class="nb"&gt;ls&lt;/span&gt; /backup/ | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;
&lt;span class="c"&gt;# Result: 43,200 small files&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Ephemeral Session Files
&lt;/h3&gt;

&lt;p&gt;PHP sessions, temporary upload files, or cache directories that grow without bounds. Each request creates a new file. Nothing cleans them up.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Check common inode traps&lt;/span&gt;
&lt;span class="nb"&gt;ls&lt;/span&gt; /var/lib/php/sessions/ | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;
&lt;span class="nb"&gt;ls&lt;/span&gt; /tmp | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Diagnosing the Culprit
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Step 1: Confirm Inode Exhaustion
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;df&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If any partition shows 100% in Use%, you've found the problem.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Find the Largest Directories
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Top-level breakdown (slow on large filesystems)&lt;/span&gt;
&lt;span class="nb"&gt;sudo du&lt;/span&gt; &lt;span class="nt"&gt;-x&lt;/span&gt; &lt;span class="nt"&gt;--max-depth&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 / 2&amp;gt;/dev/null | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-rh&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-20&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Focus on the directory with the most files, not necessarily the most bytes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Identify the Pattern
&lt;/h3&gt;

&lt;p&gt;Once you find the suspect directory, check the file characteristics:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Check if they're tiny files (the inode killer)&lt;/span&gt;
find /path/to/suspect &lt;span class="nt"&gt;-type&lt;/span&gt; f &lt;span class="nt"&gt;-size&lt;/span&gt; &lt;span class="nt"&gt;-1k&lt;/span&gt; | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;
&lt;span class="c"&gt;# If most files are under 1KB, that's your inode bomb&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Fix: Clean Up and Prevent
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Immediate Relief
&lt;/h3&gt;

&lt;p&gt;Delete the smallest, most numerous files first. Each deletion frees one inode. Prioritize:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Clear npm cache (frees thousands of tiny files)&lt;/span&gt;
npm cache clean &lt;span class="nt"&gt;--force&lt;/span&gt;

&lt;span class="c"&gt;# Rotate and compress old logs&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;journalctl &lt;span class="nt"&gt;--vacuum-time&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;3d

&lt;span class="c"&gt;# Remove temporary files older than 7 days&lt;/span&gt;
find /tmp &lt;span class="nt"&gt;-type&lt;/span&gt; f &lt;span class="nt"&gt;-mtime&lt;/span&gt; +7 &lt;span class="nt"&gt;-delete&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Long-Term Prevention
&lt;/h3&gt;

&lt;p&gt;Configure logrotate properly so old logs compress and rotate instead of piling up:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;/&lt;span class="n"&gt;var&lt;/span&gt;/&lt;span class="n"&gt;log&lt;/span&gt;/&lt;span class="n"&gt;myapp&lt;/span&gt;/*.&lt;span class="n"&gt;log&lt;/span&gt; {
    &lt;span class="n"&gt;daily&lt;/span&gt;
    &lt;span class="n"&gt;rotate&lt;/span&gt; &lt;span class="m"&gt;7&lt;/span&gt;
    &lt;span class="n"&gt;compress&lt;/span&gt;
    &lt;span class="n"&gt;delaycompress&lt;/span&gt;
    &lt;span class="n"&gt;missingok&lt;/span&gt;
    &lt;span class="n"&gt;notifempty&lt;/span&gt;
    &lt;span class="n"&gt;create&lt;/span&gt; &lt;span class="m"&gt;0640&lt;/span&gt; &lt;span class="n"&gt;www&lt;/span&gt;-&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="n"&gt;adm&lt;/span&gt;
    &lt;span class="n"&gt;postrotate&lt;/span&gt;
        &lt;span class="n"&gt;systemctl&lt;/span&gt; &lt;span class="n"&gt;reload&lt;/span&gt; &lt;span class="n"&gt;myapp&lt;/span&gt;
    &lt;span class="n"&gt;endscript&lt;/span&gt;
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then add a cron job that warns you before inodes hit critical:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0 6 * * * df -i | awk 'NR==1{next} $5 ~ /[9][0-9]%/{print "INODE ALERT: " $0}' | mail -s "Inode Usage Critical" admin@example.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  When Cleanup Isn't Enough
&lt;/h2&gt;

&lt;p&gt;Sometimes the filesystem was created with a small inode count. Check your filesystem creation parameters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;tune2fs &lt;span class="nt"&gt;-l&lt;/span&gt; /dev/sda1 | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; inode
&lt;span class="c"&gt;# Inode count:           6553600&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If your inode count is too low for your workload, you have three options:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Resize the filesystem&lt;/strong&gt; (risky, requires unmount or online resize depending on filesystem type)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Move data to a larger partition&lt;/strong&gt; with more inodes&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recreate the filesystem&lt;/strong&gt; with higher inode density (last resort)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For ext4, the inode ratio is controlled at format time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Default: 1 inode per 16KB of space&lt;/span&gt;
mke2fs &lt;span class="nt"&gt;-T&lt;/span&gt; default /dev/sda1

&lt;span class="c"&gt;# For file-heavy workloads, increase inode density&lt;/span&gt;
mke2fs &lt;span class="nt"&gt;-i&lt;/span&gt; 8192 /dev/sda1  &lt;span class="c"&gt;# 1 inode per 8KB&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Key Takeaway
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;df -h&lt;/code&gt; tells you about disk space. &lt;code&gt;df -i&lt;/code&gt; tells you about inodes. Run both.&lt;/p&gt;

&lt;p&gt;When you see "No space left on device" on a partition that clearly has free space, check inodes before you panic about disk capacity. The fix is rarely more storage — it's usually deleting thousands of tiny files you didn't know were piling up.&lt;/p&gt;




&lt;p&gt;More ops notes and server reliability guides: &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;jiunotes&lt;/a&gt; · &lt;a href="https://ops.jiunotes.top" rel="noopener noreferrer"&gt;ops notes&lt;/a&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>infrastructure</category>
      <category>linux</category>
    </item>
    <item>
      <title>DNS Debugging: Six Symptoms, One Decision Tree, and a +cd Trick That Saves an Hour</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Sat, 07 Nov 2026 09:34:49 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/dns-debugging-six-symptoms-one-decision-tree-and-a-cd-trick-that-saves-an-hour-29al</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/dns-debugging-six-symptoms-one-decision-tree-and-a-cd-trick-that-saves-an-hour-29al</guid>
      <description>&lt;p&gt;DNS gets blamed first and debugged last. The symptom is always the same — the page won't load, the API times out, the container can't reach its database — and the cause can be anywhere between your laptop's cache and a TLD server. So people run every command they remember, in random order, and learn nothing.&lt;/p&gt;

&lt;p&gt;The move that actually works is duller: &lt;strong&gt;classify the symptom first, then pick one command for that class.&lt;/strong&gt; Six symptoms cover almost every DNS failure I have run into in twenty-odd years of operations work.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;"DNS is broken"
  │
  ├─ NXDOMAIN ──────────────── The name doesn't exist, or the delegation hasn't propagated
  ├─ SERVFAIL ──────────────── Some server in the chain refused to answer
  ├─ Timeout ───────────────── The recursor is down, or port 53 is blocked on the path
  ├─ CNAME won't resolve ───── A loop, or a CNAME sitting on top of an A record
  ├─ Record won't change ───── TTL hasn't expired yet
  └─ DNSSEC fails ──────────── The signature chain is broken upstream
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Start at the top of that list, not at the top of your memory.&lt;/p&gt;

&lt;h2&gt;
  
  
  SERVFAIL: try &lt;code&gt;+cd&lt;/code&gt; before you chase anything
&lt;/h2&gt;

&lt;p&gt;This is the one worth reading the article for.&lt;/p&gt;

&lt;p&gt;SERVFAIL is DNS's version of a 500. The server refuses to answer and won't say why. Half the time it is not a broken server at all — it is DNSSEC validation failing, and the resolver is being honest about it. Two commands tell them apart:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig @8.8.8.8 example.com        &lt;span class="c"&gt;# SERVFAIL&lt;/span&gt;
dig @8.8.8.8 example.com +cd    &lt;span class="c"&gt;# validation off: NOERROR, correct IP&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the second one returns the right answer, the record is fine and the chain is broken. Look at the two ends of the chain:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig example.com DS +noall +answer        &lt;span class="c"&gt;# at the parent (registrar / TLD)&lt;/span&gt;
dig example.com DNSKEY +noall +answer    &lt;span class="c"&gt;# at the authoritative&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Different key IDs mean the registrar never synced the DS record after a key rotation. That gets fixed at the registrar, not on your server. You can spend an afternoon restarting BIND and never touch the actual problem.&lt;/p&gt;

&lt;p&gt;If both queries return SERVFAIL, the authoritative really is down and you should go chase connectivity instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  NXDOMAIN: three questions, in order
&lt;/h2&gt;

&lt;p&gt;NXDOMAIN means "this name does not exist". Ask it three times, from the outside in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig example.com                          &lt;span class="c"&gt;# your local recursor&lt;/span&gt;
dig @ns1.example.com example.com         &lt;span class="c"&gt;# the authoritative, cache bypassed&lt;/span&gt;
dig @a.gtld-servers.net example.com NS   &lt;span class="c"&gt;# the delegation at the TLD&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;NOERROR&lt;/code&gt; with a non-empty answer section → the name exists and your resolver was lying. Flush it.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;NOERROR&lt;/code&gt; with an empty answer section → the name exists but has no A/AAAA record. Add one.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;NXDOMAIN&lt;/code&gt; from the authoritative → the name genuinely doesn't exist. Check the registrar.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The classic false alarm is a domain registered ten minutes ago. The registrar console says "registered", the authoritative returns NXDOMAIN, and nothing is actually wrong: the delegation to the TLD hasn't been published yet. That has taken anywhere from 15 minutes to six hours in cases I have handled. Don't touch your resolver config; check the TLD first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeout: compare resolvers, don't guess
&lt;/h2&gt;

&lt;p&gt;One timeout tells you nothing. Three timeouts tell you where the problem is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig @8.8.8.8 example.com +time&lt;span class="o"&gt;=&lt;/span&gt;3 +tries&lt;span class="o"&gt;=&lt;/span&gt;1
dig @1.1.1.1 example.com +time&lt;span class="o"&gt;=&lt;/span&gt;3 +tries&lt;span class="o"&gt;=&lt;/span&gt;1
dig @223.5.5.5 example.com +time&lt;span class="o"&gt;=&lt;/span&gt;3 +tries&lt;span class="o"&gt;=&lt;/span&gt;1
dig example.com +time&lt;span class="o"&gt;=&lt;/span&gt;3 +tries&lt;span class="o"&gt;=&lt;/span&gt;1        &lt;span class="c"&gt;# whatever your host uses&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;All four fail → the path is blocked, or port 53 is filtered somewhere.&lt;/li&gt;
&lt;li&gt;Only the public ones fail → your internal recursor is down. Check &lt;code&gt;systemctl status systemd-resolved&lt;/code&gt; or &lt;code&gt;systemctl status bind&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Public works, internal is slow → the internal recursor is misconfigured, not dead.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When exactly one machine is affected and its neighbours on the same VLAN are fine, stop looking at the network. Check the local resolver service. A segfaulted &lt;code&gt;systemd-resolved&lt;/code&gt; looks exactly like a network fault from the application's point of view, and &lt;code&gt;sudo systemctl restart systemd-resolved&lt;/code&gt; fixes it in two seconds:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl status systemd-resolved
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart systemd-resolved
&lt;span class="nb"&gt;sudo &lt;/span&gt;resolvectl flush-caches
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  CNAME loops, and the trap next to them
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig example.com +noall +answer +comments
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A loop looks like this, and resolvers give up after eight CNAME hops:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;example.com.      300 IN CNAME www.example.com.
www.example.com.  300 IN CNAME example.com.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The neighbouring trap is quieter: a CNAME cannot share a name with any other record. This is invalid and some providers will happily save it anyway.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;example.com.      300 IN CNAME cdn.example.com.
example.com.      300 IN A     1.2.3.4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Split it into two names — &lt;code&gt;app.example.com&lt;/code&gt; pointing at &lt;code&gt;lb.example.com&lt;/code&gt;, with the A record on &lt;code&gt;lb.example.com&lt;/code&gt; — and the problem disappears.&lt;/p&gt;

&lt;h2&gt;
  
  
  "I changed the record and nothing happened"
&lt;/h2&gt;

&lt;p&gt;That's TTL, almost every time. Read it out of the answer section:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig example.com | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-20&lt;/span&gt;
&lt;span class="c"&gt;# example.com.  86400 IN A 1.2.3.4&lt;/span&gt;
&lt;span class="c"&gt;#               ^^^^^ 86400 seconds = 24 hours. You wait 24 hours.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The change procedure that avoids this, in this order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Lower the TTL to 60 seconds.&lt;/li&gt;
&lt;li&gt;Wait 60 seconds.&lt;/li&gt;
&lt;li&gt;Change the record.&lt;/li&gt;
&lt;li&gt;Wait for propagation.&lt;/li&gt;
&lt;li&gt;Put the TTL back.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Do steps 1 and 3 in the other order and the change is a no-op for a day. Then compare two independent resolvers to see who is stale:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig @8.8.8.8 example.com
dig @1.1.1.1 example.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same answer from both → wait out the TTL. Different answers → one of them is stale, flush it. Both stale → the authoritative never changed; go back to the registrar.&lt;/p&gt;

&lt;p&gt;Cache flush commands, by platform:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart systemd-resolved    &lt;span class="c"&gt;# Linux, systemd-resolved&lt;/span&gt;
&lt;span class="nb"&gt;sudo kill&lt;/span&gt; &lt;span class="nt"&gt;-SIGHUP&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;pgrep dnsmasq&lt;span class="si"&gt;)&lt;/span&gt;         &lt;span class="c"&gt;# Linux, dnsmasq&lt;/span&gt;
ipconfig /flushdns                         &lt;span class="c"&gt;# Windows&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;dscacheutil &lt;span class="nt"&gt;-flushcache&lt;/span&gt;               &lt;span class="c"&gt;# macOS&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;killall &lt;span class="nt"&gt;-HUP&lt;/span&gt; mDNSResponder            &lt;span class="c"&gt;# macOS&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  DNSSEC: read the flags, not the error message
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig example.com +dnssec +noall +comments
dig example.com +cd     +noall +comments
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An &lt;code&gt;ad&lt;/code&gt; bit in the flags line means validation succeeded. &lt;code&gt;SERVFAIL&lt;/code&gt; with &lt;code&gt;+dnssec&lt;/code&gt; and &lt;code&gt;NOERROR&lt;/code&gt; with &lt;code&gt;+cd&lt;/code&gt; means the answer is valid but unsigned somewhere up the chain. If the DS and DNSKEY key IDs don't match, the break is at the delegation — escalate to the registrar with the command output rather than rebuilding the zone.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cheat sheet
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;th&gt;What it gives you&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig +short example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The answer, one line&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig example.com A&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;A record&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig example.com AAAA&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;IPv6 record&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig example.com MX&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Mail routing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig example.com TXT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;SPF / DMARC / DKIM&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig example.com NS&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Authoritative nameservers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig @8.8.8.8 example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Ask a specific resolver&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig @ns1.example.com example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Ask the authoritative, cache bypassed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig example.com +trace&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Iterative walk from the root&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig example.com +noall +answer&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Answer section only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig example.com +dnssec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Request signatures&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dig example.com +cd&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Skip validation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;dig&lt;/code&gt; covers about 90% of this. &lt;code&gt;host&lt;/code&gt; is fine in scripts. &lt;code&gt;nslookup&lt;/code&gt; is what Windows gives you and it shows you the least — if you can &lt;code&gt;dig&lt;/code&gt;, don't &lt;code&gt;nslookup&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Still broken? Drop to packets
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;tcpdump &lt;span class="nt"&gt;-nn&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; any &lt;span class="nt"&gt;-s&lt;/span&gt; 0 port 53 &lt;span class="nt"&gt;-c&lt;/span&gt; 50
&lt;span class="nb"&gt;sudo &lt;/span&gt;tcpdump &lt;span class="nt"&gt;-nn&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; any port 53 and host 8.8.8.8
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Query leaves, no response comes back → the path or a middlebox is dropping it. Response comes back but the application never sees it → it's the local resolver, not the network.&lt;/p&gt;




&lt;p&gt;I keep a longer version of this with the full decision tree, conditional forwarding configs for BIND and dnsmasq, the Docker &lt;code&gt;127.0.0.11&lt;/code&gt; resolver behaviour, and three production post-mortems here: &lt;strong&gt;&lt;a href="https://ops.jiunotes.top/en/network/dns-troubleshoot.md" rel="noopener noreferrer"&gt;DNS troubleshooting: six-symptom decision tree&lt;/a&gt;&lt;/strong&gt;. It also covers the DNSSEC caveat I skipped above — Cloudflare, Route 53 and Namecheap each handle key rotation slightly differently.&lt;/p&gt;

&lt;p&gt;One more thing worth bookmarking: &lt;code&gt;dig example.com +trace&lt;/code&gt; is the single most informative command in this article, and almost nobody runs it before restarting something. The trace tells you which hop is broken. Restarting tells you nothing.&lt;/p&gt;

</description>
      <category>dns</category>
      <category>linux</category>
      <category>networking</category>
      <category>debugging</category>
    </item>
    <item>
      <title>Cron Not Running? 7 Reasons Your Scheduled Jobs Fail Silently</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Thu, 05 Nov 2026 22:08:24 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/cron-not-running-7-reasons-your-scheduled-jobs-fail-silently-5ap6</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/cron-not-running-7-reasons-your-scheduled-jobs-fail-silently-5ap6</guid>
      <description>&lt;h1&gt;
  
  
  Cron Not Running? 7 Reasons Your Scheduled Jobs Fail Silently
&lt;/h1&gt;

&lt;p&gt;It works when you run it manually. Then you check the logs at 3 AM and realize the cron job hasn't executed in days.&lt;/p&gt;

&lt;p&gt;This is one of the most common production nightmares I've seen in 20+ years of field engineering. The job looks fine in your head, runs fine in your terminal, but the scheduler ignores it completely. Let me walk you through the seven places I always check first.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. The PATH Trap
&lt;/h2&gt;

&lt;p&gt;This is the #1 culprit. Cron runs with a minimal PATH:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="nv"&gt;$PATH&lt;/span&gt;
&lt;span class="c"&gt;# Usually: /usr/bin:/bin&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When you run a command manually, your shell might have &lt;code&gt;/usr/local/bin&lt;/code&gt;, &lt;code&gt;/opt/bin&lt;/code&gt;, or even Python installed in a non-standard location. Cron doesn't.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# This fails if python3 is in /usr/local/bin
* * * * * python3 /opt/scripts/cleanup.py &amp;gt;&amp;gt; /var/log/cleanup.log
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Fix:&lt;/strong&gt; Use absolute paths for everything, or set PATH explicitly in the crontab:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PATH=/usr/local/bin:/usr/bin:/bin
* * * * * /usr/local/bin/python3 /opt/scripts/cleanup.py &amp;gt;&amp;gt; /var/log/cleanup.log
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  2. Environment Variables Vanish
&lt;/h2&gt;

&lt;p&gt;Your script might rely on environment variables set in &lt;code&gt;.bashrc&lt;/code&gt; or &lt;code&gt;.profile&lt;/code&gt;. Cron doesn't source these files.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# This breaks if $API_KEY is set in your .bashrc but not in cron
* * * * * /opt/scripts/fetch_data.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Fix:&lt;/strong&gt; Export variables in the crontab itself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;* * * * * export API_KEY="your_key_here" &amp;amp;&amp;amp; /opt/scripts/fetch_data.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or better yet, use a &lt;code&gt;.env&lt;/code&gt; file and source it in the script:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-a&lt;/span&gt;
&lt;span class="nb"&gt;source&lt;/span&gt; /opt/scripts/.env
&lt;span class="nb"&gt;set&lt;/span&gt; +a
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  3. Working Directory Changes
&lt;/h2&gt;

&lt;p&gt;Cron always runs from the user's home directory (&lt;code&gt;$HOME&lt;/code&gt;), not the directory where the script lives. If your script uses relative paths, it will fail silently or write to the wrong place.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# This fails if you're not in /opt/scripts when cron runs
* * * * * cd /opt/scripts &amp;amp;&amp;amp; ./process_logs.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Fix:&lt;/strong&gt; Always use absolute paths, or force the directory change at the start of the command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;* * * * * (cd /opt/scripts &amp;amp;&amp;amp; ./process_logs.sh) &amp;gt;&amp;gt; /var/log/process.log 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  4. Output Goes Nowhere (And You Miss It)
&lt;/h2&gt;

&lt;p&gt;By default, cron sends output to the user's email. If mail isn't configured, output is discarded. Your job might be failing every day and you'd never know.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Silent failure: errors go to nowhere
* * * * * /opt/scripts/broken_job.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Fix:&lt;/strong&gt; Always redirect output to a log file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;* * * * * /opt/scripts/broken_job.sh &amp;gt;&amp;gt; /var/log/cron/job.log 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check the log daily, or set up monitoring to alert on errors:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;tail&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; /var/log/cron/job.log | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;--line-buffered&lt;/span&gt; &lt;span class="s2"&gt;"ERROR"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  5. Permissions Are Wrong
&lt;/h2&gt;

&lt;p&gt;The cron user might not have permission to read the script, write to the log, or access the resources the script needs.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Check if the script is executable&lt;/span&gt;
&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-la&lt;/span&gt; /opt/scripts/broken_job.sh
&lt;span class="c"&gt;# Should show: -rwxr-xr-x&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Fix:&lt;/strong&gt; Ensure the script is executable and owned by the right user:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;chmod&lt;/span&gt; +x /opt/scripts/broken_job.sh
&lt;span class="nb"&gt;chown &lt;/span&gt;deploy:deploy /opt/scripts/broken_job.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Also check SELinux or AppArmor profiles that might block execution.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. The System Is Asleep
&lt;/h2&gt;

&lt;p&gt;If your server is a laptop, VM, or cloud instance that can hibernate, cron jobs won't run during downtime. On systemd systems, check if &lt;code&gt;suspend.target&lt;/code&gt; or &lt;code&gt;hibernate.target&lt;/code&gt; is enabled:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl status systemd-suspend.service
systemctl status systemd-hibernate.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Fix:&lt;/strong&gt; Use &lt;code&gt;systemd.path&lt;/code&gt; or &lt;code&gt;anacron&lt;/code&gt; for systems that aren't always on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Anacron runs missed jobs when the system boots&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;apt &lt;span class="nb"&gt;install &lt;/span&gt;anacron
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or set up a systemd timer that checks system uptime:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/systemd/system/myjob.timer
&lt;/span&gt;&lt;span class="nn"&gt;[Timer]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;OnBootSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;5min&lt;/span&gt;
&lt;span class="py"&gt;OnUnitActiveSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;24h&lt;/span&gt;
&lt;span class="py"&gt;AccuracySec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;1min&lt;/span&gt;
&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nn"&gt;[Install]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;WantedBy&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;timers.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  7. Hidden Syntax Errors in the Crontab
&lt;/h2&gt;

&lt;p&gt;A single extra space, a missing newline, or a malformed time field can cause cron to skip the entire line. Common mistakes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Wrong: five fields, not six
* * * * * * command   # Too many fields

# Wrong: month/day confusion
0 9 * * 1 command    # This runs at 9 AM on Mondays (1=Mon)
0 9 1 * * command    # This runs at 9 AM on the 1st of every month

# Wrong: missing newline at end of file
* * * * * /opt/scripts/final_job.sh  # Some cron implementations need a trailing newline
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Fix:&lt;/strong&gt; Validate your crontab with &lt;code&gt;crontab -l&lt;/code&gt; and review each field carefully:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# List current crontab&lt;/span&gt;
crontab &lt;span class="nt"&gt;-l&lt;/span&gt;

&lt;span class="c"&gt;# Validate syntax (GNU cron)&lt;/span&gt;
crontab &lt;span class="nt"&gt;-e&lt;/span&gt;  &lt;span class="c"&gt;# Use a linter or checker if available&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Diagnostic Checklist
&lt;/h2&gt;

&lt;p&gt;When a cron job fails silently, run this checklist in order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1. Check if cron is running&lt;/span&gt;
systemctl status cron

&lt;span class="c"&gt;# 2. Check the log&lt;/span&gt;
&lt;span class="nb"&gt;sudo tail&lt;/span&gt; &lt;span class="nt"&gt;-100&lt;/span&gt; /var/log/syslog | &lt;span class="nb"&gt;grep &lt;/span&gt;CRON

&lt;span class="c"&gt;# 3. Test the command manually with the same environment&lt;/span&gt;
&lt;span class="nb"&gt;sudo&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; deploy &lt;span class="nb"&gt;env&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="nv"&gt;HOME&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/home/deploy &lt;span class="nv"&gt;SHELL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/bin/bash &lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/usr/bin:/bin /opt/scripts/broken_job.sh

&lt;span class="c"&gt;# 4. Check permissions&lt;/span&gt;
&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-la&lt;/span&gt; /opt/scripts/broken_job.sh
&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-la&lt;/span&gt; /var/log/cron/job.log

&lt;span class="c"&gt;# 5. Verify the crontab syntax&lt;/span&gt;
crontab &lt;span class="nt"&gt;-l&lt;/span&gt; | &lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="nt"&gt;-A&lt;/span&gt;  &lt;span class="c"&gt;# Show hidden characters&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Bottom Line
&lt;/h2&gt;

&lt;p&gt;Cron is powerful but unforgiving. It doesn't send emails by default, it ignores your environment, and it runs from a fixed directory. The seven reasons above cover 90% of the silent failures I've seen in production.&lt;/p&gt;

&lt;p&gt;Next time your scheduled job vanishes at 3 AM, check these seven spots before you pull your hair out.&lt;/p&gt;




&lt;p&gt;More ops notes and server reliability guides: &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;jiunotes&lt;/a&gt; · &lt;a href="https://ops.jiunotes.top" rel="noopener noreferrer"&gt;ops notes&lt;/a&gt;&lt;/p&gt;

</description>
      <category>automation</category>
      <category>debugging</category>
      <category>devops</category>
      <category>linux</category>
    </item>
    <item>
      <title>EMFILE on Production: A Field Engineer's Guide to File Descriptor Limits</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Wed, 04 Nov 2026 22:02:51 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/emfile-on-production-a-field-engineers-guide-to-file-descriptor-limits-611</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/emfile-on-production-a-field-engineers-guide-to-file-descriptor-limits-611</guid>
      <description>&lt;h1&gt;
  
  
  EMFILE on Production: A Field Engineer's Guide to File Descriptor Limits
&lt;/h1&gt;

&lt;p&gt;It hit at 2 AM. Not with a dramatic stack trace — just a sudden, silent drop in request throughput. Nginx was still serving pages, but new connections weren't being accepted. The error log told the story in one line: &lt;code&gt;accept4() failed (24: Too many open files)&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;If you've ever seen this on a production server, you know the feeling: the panic of a midnight page, the half-remembered commands, the desperate hunt for which layer of limits is choking your process.&lt;/p&gt;

&lt;p&gt;I've seen this error across dozens of servers — Node.js APIs, Python scrapers, Java microservices, Go web servers. Each time, the fix was the same: understand the three layers of Linux file descriptor limits, and tune them correctly. Let me walk you through what EMFILE really means, how to diagnose it fast, and the exact fixes that work in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  What "File" Really Means in Linux
&lt;/h2&gt;

&lt;p&gt;Here's the first thing most engineers miss: in Linux, &lt;strong&gt;a "file" is anything that can be read from or written to&lt;/strong&gt;. This includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Regular files (logs, configs, data)&lt;/li&gt;
&lt;li&gt;Network sockets (TCP connections, HTTP requests)&lt;/li&gt;
&lt;li&gt;Pipes and FIFOs&lt;/li&gt;
&lt;li&gt;Device files (&lt;code&gt;/dev/null&lt;/code&gt;, TTYs)&lt;/li&gt;
&lt;li&gt;Inotify watches&lt;/li&gt;
&lt;li&gt;epoll instances&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So when your web server handles 10,000 concurrent connections, it's opening 10,000 "files." When a log rotation script opens old and new log files simultaneously, that's two more. The numbers add up fast.&lt;/p&gt;

&lt;p&gt;The kernel assigns each open resource a &lt;strong&gt;file descriptor&lt;/strong&gt; — a small non-negative integer. The first three (0, 1, 2) are reserved for stdin, stdout, and stderr. Everything else starts at 3 and counts up.&lt;/p&gt;

&lt;p&gt;By default, most Linux distributions set the per-process soft limit to &lt;strong&gt;1024&lt;/strong&gt;. For a desktop app, that's plenty. For a server under load? It's a trap waiting to spring.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Three Layers of Limits
&lt;/h2&gt;

&lt;p&gt;Think of file descriptor limits as a system of nested buckets. All three must be high enough for your workload.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 1: Per-Process Soft Limit
&lt;/h3&gt;

&lt;p&gt;This is what &lt;code&gt;ulimit -n&lt;/code&gt; shows you. It's the active ceiling for a single process.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;ulimit&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt;
1024
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A non-root user can raise this limit on their own, but only up to the hard limit. Once raised, it applies to all child processes spawned from that shell.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The catch:&lt;/strong&gt; changing this limit in a login shell does nothing for a daemon that started before your change. The limit is copied into the process at fork time. You must restart the service.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 2: Per-Process Hard Limit
&lt;/h3&gt;

&lt;p&gt;This is the ceiling that only root can raise. It's set in &lt;code&gt;/etc/security/limits.conf&lt;/code&gt; and applies to new sessions.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;ulimit&lt;/span&gt; &lt;span class="nt"&gt;-Hn&lt;/span&gt;
1024
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If soft and hard are equal, no user can raise the limit further without root. The common production setup sets both to 65535.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 3: System-Wide Limit
&lt;/h3&gt;

&lt;p&gt;Even if every process can open 65535 files, the kernel has an overall cap on total file descriptors across all processes. Check it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/sys/fs/file-max
823008
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is rarely the bottleneck unless you're running thousands of high-concurrency services on a single box. For most setups, 2 million (&lt;code&gt;2097152&lt;/code&gt;) is a safe target.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/sys/fs/file-nr
18432   0   823008
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The three numbers: currently allocated FDs, unused, and the max. If the first number is approaching the third, you need to raise &lt;code&gt;file-max&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Diagnosing EMFILE in Production
&lt;/h2&gt;

&lt;p&gt;When the alert fires, you need answers fast. Here's the diagnostic workflow:&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Confirm Which Process Is Affected
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Find the PID of your service&lt;/span&gt;
pgrep &lt;span class="nt"&gt;-f&lt;/span&gt; &lt;span class="s2"&gt;"node|nginx|python|java"&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-5&lt;/span&gt;

&lt;span class="c"&gt;# Check its current limits&lt;/span&gt;
&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/&amp;lt;PID&amp;gt;/limits | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-A1&lt;/span&gt; &lt;span class="s2"&gt;"open files"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You'll see output like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Max open files            1024                1024                files
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both soft and hard are 1024 — that's your smoking gun.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Count Current Open File Descriptors
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;ls&lt;/span&gt; /proc/&amp;lt;PID&amp;gt;/fd | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If this is near 1024, the process is genuinely at its limit. If it's well below (say, 300), something else is going on — possibly a misconfigured service that's not actually hitting the limit, or a different error masquerading as EMFILE.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Identify What's Consuming FDs
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# See what types of files are open&lt;/span&gt;
&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; /proc/&amp;lt;PID&amp;gt;/fd | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-30&lt;/span&gt;

&lt;span class="c"&gt;# Count by type&lt;/span&gt;
&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; /proc/&amp;lt;PID&amp;gt;/fd | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'{print $9}'&lt;/span&gt; | &lt;span class="nb"&gt;sed&lt;/span&gt; &lt;span class="s1"&gt;'s|/dev/tcp/||'&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; | &lt;span class="nb"&gt;uniq&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-rn&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-20&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Common patterns:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Hundreds of &lt;code&gt;/dev/tcp/*&lt;/code&gt; entries → network connections (normal for a busy server)&lt;/li&gt;
&lt;li&gt;Many log files open → logging misconfiguration&lt;/li&gt;
&lt;li&gt;Thousands of &lt;code&gt;/tmp/*&lt;/code&gt; → temp file leak&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;eventfd&lt;/code&gt;, &lt;code&gt;timerfd&lt;/code&gt; → modern I/O event loops (epoll/kqueue style)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Step 4: Check for FD Leaks
&lt;/h3&gt;

&lt;p&gt;If the FD count climbs steadily over 30 minutes without corresponding traffic growth, you likely have a leak:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Snap twice, 30 seconds apart&lt;/span&gt;
&lt;span class="nb"&gt;sleep &lt;/span&gt;30 &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;ls&lt;/span&gt; /proc/&amp;lt;PID&amp;gt;/fd | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A steady climb = leak. You'll need to identify the code path that opens but doesn't close. Common culprits: unclosed database connections, file handles in loops without &lt;code&gt;finally&lt;/code&gt;/&lt;code&gt;defer&lt;/code&gt;, socket connections in retry logic.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Fix: Raising Limits Correctly
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Quick Fix: Current Session
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Raise soft limit to 65535 (must be &amp;lt;= hard limit)&lt;/span&gt;
&lt;span class="nb"&gt;ulimit&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; 65535

&lt;span class="c"&gt;# Verify&lt;/span&gt;
&lt;span class="nb"&gt;ulimit&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This works immediately for the current shell and its children. &lt;strong&gt;But it resets on logout.&lt;/strong&gt; Don't use this as your permanent solution — it's a bridge while you configure the proper fix.&lt;/p&gt;

&lt;h3&gt;
  
  
  Permanent Fix: /etc/security/limits.conf
&lt;/h3&gt;

&lt;p&gt;Edit as root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;nano /etc/security/limits.conf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Add at the end:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;* &lt;span class="n"&gt;soft&lt;/span&gt; &lt;span class="n"&gt;nofile&lt;/span&gt; &lt;span class="m"&gt;65535&lt;/span&gt;
* &lt;span class="n"&gt;hard&lt;/span&gt; &lt;span class="n"&gt;nofile&lt;/span&gt; &lt;span class="m"&gt;65535&lt;/span&gt;
&lt;span class="n"&gt;root&lt;/span&gt; &lt;span class="n"&gt;soft&lt;/span&gt; &lt;span class="n"&gt;nofile&lt;/span&gt; &lt;span class="m"&gt;65535&lt;/span&gt;
&lt;span class="n"&gt;root&lt;/span&gt; &lt;span class="n"&gt;hard&lt;/span&gt; &lt;span class="n"&gt;nofile&lt;/span&gt; &lt;span class="m"&gt;65535&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or, to set per-user limits:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;deploy soft nofile 65535
deploy hard nofile 65535
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Critical:&lt;/strong&gt; Users must log out and back in for this to take effect. Service managers reading this file (like systemd with PAM) will pick it up on next start.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Systemd Trap (Where Most People Get Stuck)
&lt;/h3&gt;

&lt;p&gt;Here's the gotcha that cost me two hours once: &lt;strong&gt;systemd ignores &lt;code&gt;/etc/security/limits.conf&lt;/code&gt; by default.&lt;/strong&gt; If your service runs as a systemd unit, the limits you set in &lt;code&gt;limits.conf&lt;/code&gt; won't apply.&lt;/p&gt;

&lt;p&gt;Instead, create an override:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl edit nginx
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This opens an editor with an empty drop-in file. Add:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;LimitNOFILE&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;65535&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Save and exit, then:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl daemon-reload
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart nginx
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Verify the running process picked it up:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;PID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;pgrep &lt;span class="nt"&gt;-f&lt;/span&gt; nginx | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-1&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/limits | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="s2"&gt;"open files"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Max open files            65535               65535               files
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can also preview what systemd will apply without restarting:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl show &lt;span class="nt"&gt;-p&lt;/span&gt; LimitNOFILE nginx
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  System-Wide Kernel Limit
&lt;/h3&gt;

&lt;p&gt;If you're running many high-FD processes (a VPS with multiple Node.js services, for example), raise the kernel ceiling:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Check current value&lt;/span&gt;
&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/sys/fs/file-max

&lt;span class="c"&gt;# Temporarily raise (resets on reboot)&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;sysctl &lt;span class="nt"&gt;-w&lt;/span&gt; fs.file-max&lt;span class="o"&gt;=&lt;/span&gt;2097152

&lt;span class="c"&gt;# Permanently add to sysctl config&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"fs.file-max = 2097152"&lt;/span&gt; | &lt;span class="nb"&gt;sudo tee&lt;/span&gt; &lt;span class="nt"&gt;-a&lt;/span&gt; /etc/sysctl.conf
&lt;span class="nb"&gt;sudo &lt;/span&gt;sysctl &lt;span class="nt"&gt;-p&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;2 million FDs system-wide handles ~30 processes at 65535 each with room to spare.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Verification Mindset
&lt;/h2&gt;

&lt;p&gt;Here's the golden rule: &lt;strong&gt;never trust a config file. Trust &lt;code&gt;/proc&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Config files are promises. &lt;code&gt;/proc&lt;/code&gt; is reality. After every change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1. Find the running process&lt;/span&gt;
&lt;span class="nv"&gt;PID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;pgrep &lt;span class="nt"&gt;-f&lt;/span&gt; &lt;span class="s2"&gt;"your-service"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;

&lt;span class="c"&gt;# 2. Check actual limits&lt;/span&gt;
&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/limits | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="s2"&gt;"open files"&lt;/span&gt;

&lt;span class="c"&gt;# 3. Check actual usage&lt;/span&gt;
&lt;span class="nb"&gt;ls&lt;/span&gt; /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/fd | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;

&lt;span class="c"&gt;# 4. Cross-reference&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Usage: &lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;ls&lt;/span&gt; /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/fd | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt; / Limit: &lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/limits | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="s1"&gt;'open files'&lt;/span&gt; | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'{print $NF}'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If usage is climbing toward the limit even after your fix, you haven't solved the problem — you've only delayed the next crash. Go back to Step 4 and hunt for the leak.&lt;/p&gt;

&lt;h2&gt;
  
  
  Special Cases
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Alpine Linux / musl
&lt;/h3&gt;

&lt;p&gt;Alpine doesn't use PAM. Editing &lt;code&gt;/etc/security/limits.conf&lt;/code&gt; has zero effect. Instead, set limits in your entrypoint script:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Dockerfile&lt;/span&gt;
RUN apk add &lt;span class="nt"&gt;--no-cache&lt;/span&gt; util-linux
&lt;span class="c"&gt;# ...&lt;/span&gt;
ENTRYPOINT &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"sh"&lt;/span&gt;, &lt;span class="s2"&gt;"-c"&lt;/span&gt;, &lt;span class="s2"&gt;"ulimit -n 65535 &amp;amp;&amp;amp; exec your-app"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or use a sidecar/init container that calls &lt;code&gt;prlimit&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  macOS / launchd
&lt;/h3&gt;

&lt;p&gt;macOS uses &lt;code&gt;launchd&lt;/code&gt;, not systemd. GUI apps and launchd services ignore &lt;code&gt;ulimit&lt;/code&gt;. Set limits via a plist:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight xml"&gt;&lt;code&gt;&lt;span class="cp"&gt;&amp;lt;?xml version="1.0" encoding="UTF-8"?&amp;gt;&lt;/span&gt;
&lt;span class="cp"&gt;&amp;lt;!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;plist&lt;/span&gt; &lt;span class="na"&gt;version=&lt;/span&gt;&lt;span class="s"&gt;"1.0"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;dict&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;key&amp;gt;&lt;/span&gt;Label&lt;span class="nt"&gt;&amp;lt;/key&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;string&amp;gt;&lt;/span&gt;limit.maxfiles&lt;span class="nt"&gt;&amp;lt;/string&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;key&amp;gt;&lt;/span&gt;ProgramArguments&lt;span class="nt"&gt;&amp;lt;/key&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;array&amp;gt;&lt;/span&gt;
        &lt;span class="nt"&gt;&amp;lt;string&amp;gt;&lt;/span&gt;launchctl&lt;span class="nt"&gt;&amp;lt;/string&amp;gt;&lt;/span&gt;
        &lt;span class="nt"&gt;&amp;lt;string&amp;gt;&lt;/span&gt;limit&lt;span class="nt"&gt;&amp;lt;/string&amp;gt;&lt;/span&gt;
        &lt;span class="nt"&gt;&amp;lt;string&amp;gt;&lt;/span&gt;maxfiles&lt;span class="nt"&gt;&amp;lt;/string&amp;gt;&lt;/span&gt;
        &lt;span class="nt"&gt;&amp;lt;string&amp;gt;&lt;/span&gt;65536&lt;span class="nt"&gt;&amp;lt;/string&amp;gt;&lt;/span&gt;
        &lt;span class="nt"&gt;&amp;lt;string&amp;gt;&lt;/span&gt;200000&lt;span class="nt"&gt;&amp;lt;/string&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;/array&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;key&amp;gt;&lt;/span&gt;RunAtLoad&lt;span class="nt"&gt;&amp;lt;/key&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;true/&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/dict&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/plist&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Save to &lt;code&gt;/Library/LaunchDaemons/limit.maxfiles.plist&lt;/code&gt; and load it. This trips up almost everyone who develops locally on Mac and deploys to Linux.&lt;/p&gt;

&lt;h3&gt;
  
  
  Containers and cgroups
&lt;/h3&gt;

&lt;p&gt;Docker containers inherit limits from the host by default. To set custom limits:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker run &lt;span class="nt"&gt;--ulimit&lt;/span&gt; &lt;span class="nv"&gt;nofile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;65535:65535 your-image
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or in Kubernetes, set it in the pod spec:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;securityContext&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;sysctls&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;fs.file-max&lt;/span&gt;
    &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2097152"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note: cgroups v2 may introduce a &lt;code&gt;pids.max&lt;/code&gt; cap. If "too many open files" coincides with &lt;code&gt;fork&lt;/code&gt; failures, check the PID cgroup, not just &lt;code&gt;nofile&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prevention: Monitoring That Actually Works
&lt;/h2&gt;

&lt;p&gt;Raising limits is reactive. Prevention is better. Set up monitoring that catches FD exhaustion before it becomes an outage:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# One-liner for your monitoring agent&lt;/span&gt;
watch &lt;span class="nt"&gt;-n&lt;/span&gt; 60 &lt;span class="s1"&gt;'echo "PID: $(pgrep -f your-service) | Used: $(ls /proc/$(pgrep -f your-service)/fd | wc -l) / Limit: $(cat /proc/$(pgrep -f your-service)/limits | grep open | awk "{print \$(NF-1)}")"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or integrate with Prometheus:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# node_exporter collects fd_usage automatically&lt;/span&gt;
&lt;span class="c1"&gt;# Alert when usage &amp;gt; 80% of limit&lt;/span&gt;
&lt;span class="na"&gt;groups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;filedescriptors&lt;/span&gt;
    &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HighFileDescriptorUsage&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;fd_used / fd_limit &amp;gt; &lt;/span&gt;&lt;span class="m"&gt;0.8&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Set alerts at 70% (warning) and 85% (critical). A slow leak gives you days of advance notice; a sudden spike gives you minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Bottom Line
&lt;/h2&gt;

&lt;p&gt;EMFILE is one of those errors that feels arcane until you understand the three layers. Once you do, it's straightforward:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Diagnose&lt;/strong&gt;: Check &lt;code&gt;/proc/&amp;lt;PID&amp;gt;/limits&lt;/code&gt; and count &lt;code&gt;/proc/&amp;lt;PID&amp;gt;/fd&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fix&lt;/strong&gt;: Raise limits at the right layer (systemd override, not just &lt;code&gt;limits.conf&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify&lt;/strong&gt;: Trust &lt;code&gt;/proc&lt;/code&gt;, not config files&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prevent&lt;/strong&gt;: Monitor the ratio of used-to-limit before it hits 100%&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The next time your phone buzzes at 2 AM with an EMFILE alert, you'll know exactly where to look — and more importantly, you'll know how to explain to your team that this wasn't a mystery, just a configuration that hadn't caught up with the workload.&lt;/p&gt;




&lt;p&gt;More ops notes and server reliability guides: &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;jiunotes&lt;/a&gt; · &lt;a href="https://ops.jiunotes.top" rel="noopener noreferrer"&gt;ops notes&lt;/a&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>infrastructure</category>
      <category>linux</category>
      <category>sre</category>
    </item>
    <item>
      <title>Stop Using Passwords for SSH: The Key-Based Auth Setup Guide</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Wed, 04 Nov 2026 01:04:45 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/stop-using-passwords-for-ssh-the-key-based-auth-setup-guide-bcb</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/stop-using-passwords-for-ssh-the-key-based-auth-setup-guide-bcb</guid>
      <description>&lt;p&gt;Every sysadmin knows the drill. You log into a dozen servers a day, each with a password you half-remember. Some are memorable; some you've reset three times this month because you wrote them on a sticky note and lost the note. And every time you type that password, you wonder: is this secure? Should I be using something better?&lt;/p&gt;

&lt;p&gt;The answer is yes. Key-based authentication is not just better — it's the baseline for any serious server management. Passwords can be brute-forced. Keys (used correctly) cannot. Here's how to set it up, the mistakes people make, and the one thing most guides forget to mention.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Keys Beat Passwords Every Time
&lt;/h2&gt;

&lt;p&gt;A strong password is still guessable given enough time and compute. A 4096-bit RSA key or an Ed25519 key? The math doesn't work that way. The computational effort to brute-force a proper SSH key exceeds the heat death of the universe.&lt;/p&gt;

&lt;p&gt;But keys aren't magic. A misconfigured key setup can be worse than a password — you hand someone a master key with no door lock. That's why the setup matters more than the algorithm choice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Generate Your Key Pair
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-keygen &lt;span class="nt"&gt;-t&lt;/span&gt; ed25519 &lt;span class="nt"&gt;-C&lt;/span&gt; &lt;span class="s2"&gt;"your_email@example.com"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ed25519 is the current recommendation. It's faster than RSA, shorter keys for equivalent security, and less prone to implementation bugs. If you're working with legacy systems that don't support it yet, fall back to RSA 4096:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-keygen &lt;span class="nt"&gt;-t&lt;/span&gt; rsa &lt;span class="nt"&gt;-b&lt;/span&gt; 4096 &lt;span class="nt"&gt;-C&lt;/span&gt; &lt;span class="s2"&gt;"your_email@example.com"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;-C&lt;/code&gt; flag adds a comment (usually your email) so you can identify the key later. Don't skip this — you'll thank yourself when you have twenty keys and need to figure out which one belongs to which server.&lt;/p&gt;

&lt;p&gt;When prompted for a passphrase, &lt;strong&gt;use one&lt;/strong&gt;. A key without a passphrase is a master key with no lock. Yes, it's slightly inconvenient to type it. No, that inconvenience is the whole point.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Get the Public Key onto the Server
&lt;/h2&gt;

&lt;p&gt;The easy way:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-copy-id user@server
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This copies your public key (&lt;code&gt;~/.ssh/id_ed25519.pub&lt;/code&gt;) to the server's &lt;code&gt;~/.ssh/authorized_keys&lt;/code&gt; file and sets correct permissions.&lt;/p&gt;

&lt;p&gt;The manual way (if ssh-copy-id isn't available):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; ~/.ssh/id_ed25519.pub | ssh user@server &lt;span class="s2"&gt;"mkdir -p ~/.ssh &amp;amp;&amp;amp; cat &amp;gt;&amp;gt; ~/.ssh/authorized_keys &amp;amp;&amp;amp; chmod 600 ~/.ssh/authorized_keys"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both methods do the same thing. Pick whichever feels right.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Harden the Server Configuration
&lt;/h2&gt;

&lt;p&gt;Logging in with a key is only half the battle. You need to tell the server to reject password auth entirely:&lt;/p&gt;

&lt;p&gt;Edit &lt;code&gt;/etc/ssh/sshd_config&lt;/code&gt; on the server:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ssh"&gt;&lt;code&gt;&lt;span class="k"&gt;PasswordAuthentication&lt;/span&gt; &lt;span class="no"&gt;no&lt;/span&gt;
&lt;span class="k"&gt;PubkeyAuthentication&lt;/span&gt; &lt;span class="no"&gt;yes&lt;/span&gt;
&lt;span class="k"&gt;ChallengeResponseAuthentication&lt;/span&gt; &lt;span class="no"&gt;no&lt;/span&gt;
&lt;span class="k"&gt;UsePAM&lt;/span&gt; &lt;span class="no"&gt;no&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then restart SSH:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart sshd
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Critical:&lt;/strong&gt; Before you disable password auth, verify you can actually log in with your key. Close your current session, open a new one, and test. If you lock yourself out, you'll need physical access or a console to fix it — and that's a bad day.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: Manage Your Keys Properly
&lt;/h2&gt;

&lt;p&gt;Most people create one key and use it everywhere. This is convenient until that one key gets compromised, and then you have to rotate it across every server you own.&lt;/p&gt;

&lt;p&gt;Better approach: create separate keys for different contexts.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-keygen &lt;span class="nt"&gt;-t&lt;/span&gt; ed25519 &lt;span class="nt"&gt;-C&lt;/span&gt; &lt;span class="s2"&gt;"work-laptop"&lt;/span&gt;
ssh-keygen &lt;span class="nt"&gt;-t&lt;/span&gt; ed25519 &lt;span class="nt"&gt;-C&lt;/span&gt; &lt;span class="s2"&gt;"home-server"&lt;/span&gt;
ssh-keygen &lt;span class="nt"&gt;-t&lt;/span&gt; ed25519 &lt;span class="nt"&gt;-C&lt;/span&gt; &lt;span class="s2"&gt;"ci-cd-pipeline"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then configure &lt;code&gt;~/.ssh/config&lt;/code&gt; to use the right key for the right host:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ssh"&gt;&lt;code&gt;&lt;span class="k"&gt;Host&lt;/span&gt; work-*
    &lt;span class="k"&gt;IdentityFile&lt;/span&gt; ~/.ssh/id_work_laptop

&lt;span class="k"&gt;Host&lt;/span&gt; home-*
    &lt;span class="k"&gt;IdentityFile&lt;/span&gt; ~/.ssh/id_home_server

&lt;span class="k"&gt;Host&lt;/span&gt; ci-*
    &lt;span class="k"&gt;IdentityFile&lt;/span&gt; ~/.ssh/id_ci_cd
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This keeps your keys compartmentalized. If your CI/CD key leaks, you only rotate that one — not your personal access to every server.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Mistake Everyone Makes
&lt;/h2&gt;

&lt;p&gt;People forget to set permissions on their private key. If &lt;code&gt;~/.ssh/id_ed25519&lt;/code&gt; is world-readable, SSH will refuse to use it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Permissions 0644 for '~/.ssh/id_ed25519' are too open.
It is required that your private key files are NOT accessible by others.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Fix it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;chmod &lt;/span&gt;600 ~/.ssh/id_ed25519
&lt;span class="nb"&gt;chmod &lt;/span&gt;700 ~/.ssh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These permissions are non-negotiable. SSH enforces them strictly.&lt;/p&gt;

&lt;h2&gt;
  
  
  When Keys Aren't Enough
&lt;/h2&gt;

&lt;p&gt;Key-based auth solves authentication. It doesn't solve authorization. Just because someone has a valid key doesn't mean they should have root on your production database.&lt;/p&gt;

&lt;p&gt;Pair key-based auth with:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Limited sudo&lt;/strong&gt; in &lt;code&gt;/etc/sudoers&lt;/code&gt; — give users only the commands they need&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Jump hosts&lt;/strong&gt; — force all access through a bastion server&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Agent forwarding caution&lt;/strong&gt; — forwarding your SSH agent to a remote server means that server can use your keys. Trust the remote, or don't forward.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A Final Thought
&lt;/h2&gt;

&lt;p&gt;SSH key-based authentication isn't complicated. It's just slightly more work than typing a password. That extra work is what makes it secure.&lt;/p&gt;

&lt;p&gt;If you're still using passwords in 2026, you're not being careful — you're being lazy. And in security, laziness is the most expensive trait you can have.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Tags: ssh, security, linux, devops, tutorial&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;More field-tested ops notes and server security guides: &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;jiunotes&lt;/a&gt; · &lt;a href="https://ops.jiunotes.top" rel="noopener noreferrer"&gt;ops notes&lt;/a&gt;&lt;/p&gt;

</description>
      <category>authentication</category>
      <category>devops</category>
      <category>linux</category>
      <category>security</category>
    </item>
    <item>
      <title>What's My Chinese Zodiac Sign? A Quick, Accurate Way to Find It</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Mon, 02 Nov 2026 22:38:28 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/whats-my-chinese-zodiac-sign-a-quick-accurate-way-to-find-it-2na3</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/whats-my-chinese-zodiac-sign-a-quick-accurate-way-to-find-it-2na3</guid>
      <description>&lt;h1&gt;
  
  
  What's My Chinese Zodiac Sign? A Quick, Accurate Way to Find It
&lt;/h1&gt;

&lt;p&gt;If you've ever chatted with a Chinese friend and heard them say, "I'm a Tiger" or "Oh, you're a Snake, no wonder," you might have wondered how they figured that out so fast. It's not a guess. The Chinese zodiac is one of the easiest, most fun parts of Chinese culture to learn — and once you know the trick, you can place almost anyone in under five seconds.&lt;/p&gt;

&lt;p&gt;Here's the one thing most Western guides get wrong: the zodiac is NOT based on January 1. It follows the lunar calendar, and the year flips on Chinese New Year (Spring Festival), which falls somewhere between late January and late February each year. If you were born in January or February, you need to check the exact date of that year's New Year before you claim a sign.&lt;/p&gt;

&lt;h2&gt;
  
  
  The One Rule That Fixes Almost Everything
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Born in March through December? Your zodiac sign is simply the one for that calendar year.&lt;/li&gt;
&lt;li&gt;Born in January or February? Look up when Chinese New Year fell that year. If your birthday came before it, you actually belong to the previous year's animal.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, someone born on February 5, 2024 — before the Year of the Dragon began on February 10 — is actually a Rabbit, not a Dragon. This small detail catches a lot of people off guard, and it's the difference between an accurate answer and a confused one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Chinese Zodiac Year Chart (1996–2026)
&lt;/h2&gt;

&lt;p&gt;Use this quick reference to find your sign. Each animal repeats every 12 years, so if you don't see your exact year, add or subtract 12.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Year&lt;/th&gt;
&lt;th&gt;Animal&lt;/th&gt;
&lt;th&gt;Year&lt;/th&gt;
&lt;th&gt;Animal&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1996&lt;/td&gt;
&lt;td&gt;Rat&lt;/td&gt;
&lt;td&gt;2012&lt;/td&gt;
&lt;td&gt;Dragon&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1997&lt;/td&gt;
&lt;td&gt;Ox&lt;/td&gt;
&lt;td&gt;2013&lt;/td&gt;
&lt;td&gt;Snake&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1998&lt;/td&gt;
&lt;td&gt;Tiger&lt;/td&gt;
&lt;td&gt;2014&lt;/td&gt;
&lt;td&gt;Horse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1999&lt;/td&gt;
&lt;td&gt;Rabbit&lt;/td&gt;
&lt;td&gt;2015&lt;/td&gt;
&lt;td&gt;Goat&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2000&lt;/td&gt;
&lt;td&gt;Dragon&lt;/td&gt;
&lt;td&gt;2016&lt;/td&gt;
&lt;td&gt;Monkey&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2001&lt;/td&gt;
&lt;td&gt;Snake&lt;/td&gt;
&lt;td&gt;2017&lt;/td&gt;
&lt;td&gt;Rooster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2002&lt;/td&gt;
&lt;td&gt;Horse&lt;/td&gt;
&lt;td&gt;2018&lt;/td&gt;
&lt;td&gt;Dog&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2003&lt;/td&gt;
&lt;td&gt;Goat&lt;/td&gt;
&lt;td&gt;2019&lt;/td&gt;
&lt;td&gt;Pig&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2004&lt;/td&gt;
&lt;td&gt;Monkey&lt;/td&gt;
&lt;td&gt;2020&lt;/td&gt;
&lt;td&gt;Rat&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2005&lt;/td&gt;
&lt;td&gt;Rooster&lt;/td&gt;
&lt;td&gt;2021&lt;/td&gt;
&lt;td&gt;Ox&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2006&lt;/td&gt;
&lt;td&gt;Dog&lt;/td&gt;
&lt;td&gt;2022&lt;/td&gt;
&lt;td&gt;Tiger&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2007&lt;/td&gt;
&lt;td&gt;Pig&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;td&gt;Rabbit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2008&lt;/td&gt;
&lt;td&gt;Rat&lt;/td&gt;
&lt;td&gt;2024&lt;/td&gt;
&lt;td&gt;Dragon&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2009&lt;/td&gt;
&lt;td&gt;Ox&lt;/td&gt;
&lt;td&gt;2025&lt;/td&gt;
&lt;td&gt;Snake&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2010&lt;/td&gt;
&lt;td&gt;Tiger&lt;/td&gt;
&lt;td&gt;2026&lt;/td&gt;
&lt;td&gt;Horse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2011&lt;/td&gt;
&lt;td&gt;Rabbit&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Stuck between two years? Double-check the Chinese New Year dates: 2024 began February 10, 2025 began January 29, and 2026 began February 17. Everything before that date in the same calendar year belongs to the previous animal.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 12 Animals at a Glance
&lt;/h2&gt;

&lt;p&gt;Here's a friendly, memory-friendly summary of each sign so you can use them in conversation.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rat&lt;/strong&gt; — Quick-witted and adaptable. The rat survives by being clever, and people born under it are often seen as resourceful problem-solvers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ox&lt;/strong&gt; — Steady, patient, and dependable. The ox is the animal that works the field all day without complaining, and the personality usually matches.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tiger&lt;/strong&gt; — Bold, confident, and a little fearless. Tigers are natural leaders who love a challenge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rabbit&lt;/strong&gt; — Gentle, kind, and calm. Rabbits tend to avoid conflict and are known for being easy to get along with.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dragon&lt;/strong&gt; — Charismatic, energetic, and larger than life. The dragon is the only mythical creature in the zodiac, and it carries a sense of confidence and ambition.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Snake&lt;/strong&gt; — Thoughtful, wise, and quietly observant. Snakes think before they act and often have a mysterious, clever streak.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Horse&lt;/strong&gt; — Lively, outgoing, and free-spirited. Horses love movement, travel, and being around people.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Goat&lt;/strong&gt; — Soft-hearted, creative, and peace-loving. Goats are often artistic and care deeply about harmony.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Monkey&lt;/strong&gt; — Smart, playful, and quick to learn. Monkeys are curious and love to entertain.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rooster&lt;/strong&gt; — Punctual, honest, and observant. Roosters are the timekeepers — they notice details and aren't afraid to say what they think.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dog&lt;/strong&gt; — Loyal, honest, and protective. Dogs are the friends you want in your corner, always reliable and true.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pig&lt;/strong&gt; — Generous, easygoing, and sincere. Pigs enjoy the good things in life and are known for their warm, honest nature.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Of course, nobody is defined by one animal. Think of these as conversation starters and gentle personality hints, not a full description of who someone is.&lt;/p&gt;

&lt;h2&gt;
  
  
  Benmingnian: Your Zodiac "Year of Fate"
&lt;/h2&gt;

&lt;p&gt;Every 12 years, you hit your own zodiac year again — 12, 24, 36, 48, and so on. This is called &lt;strong&gt;benmingnian&lt;/strong&gt; (本命年), your "fate year" or "birth year return." Chinese tradition treats it as a year that needs a little extra care, because the zodiac animal is said to be in a sensitive position.&lt;/p&gt;

&lt;p&gt;The most visible custom is wearing red. Red is considered an auspicious, protective color, so many Chinese people wear red underwear, a red string bracelet, or a red belt throughout their benmingnian year. You'll often see red cords tied around wrists or ankles, sometimes with a small jade or gold charm. Some people keep it low-key; others go all in with a red outfit for New Year's Eve.&lt;/p&gt;

&lt;p&gt;Why red? In Chinese culture, red symbolizes luck, vitality, and warding off bad energy. It's the same reason you see red lanterns, red envelopes, and red couplets everywhere during Spring Festival. If your Chinese friend asks what your sign is and then laughs, "That's your benmingnian!" — they're just being playful, and maybe gently suggesting you grab a red bracelet.&lt;/p&gt;

&lt;h2&gt;
  
  
  Using Zodiac Signs in Conversation
&lt;/h2&gt;

&lt;p&gt;This is where the zodiac becomes genuinely useful. It's a fantastic icebreaker, especially with people you've just met.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ask someone their sign when you're at a dinner table or a group outing. It's a low-stakes, friendly question that almost always gets a smile.&lt;/li&gt;
&lt;li&gt;Share your own sign and a trait that fits you. "I'm a Horse, so apparently I'm supposed to be free-spirited — I think I'll take that."&lt;/li&gt;
&lt;li&gt;If you know the 12-year cycle, you can often guess someone's general age bracket. A person born in the Year of the Dragon is either around 12, 24, 36, or 48 — which narrows things down nicely without being pushy.&lt;/li&gt;
&lt;li&gt;Use it to remember birthdays. If you know a friend is a Tiger, and you know the last Tiger year, you can roughly figure out when their birthday is coming up.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The best part is that you don't need to memorize anything complicated. A simple chart like the one above, or even just remembering the 12 animals in order, is enough to participate.&lt;/p&gt;

&lt;h2&gt;
  
  
  A Reasonable Perspective
&lt;/h2&gt;

&lt;p&gt;It's worth saying plainly: the zodiac is a cultural tradition and a social lubricant, not a fortune-telling tool. Chinese people enjoy it the way anyone enjoys a horoscope — it's fun, it's a shared reference point, and it adds a little color to everyday conversation. Nobody serious is making life decisions based on which animal they were born under. So treat it lightly, have fun with it, and you'll be joining in the conversation the right way.&lt;/p&gt;

&lt;p&gt;The next time someone asks you what your sign is, you'll have a quick, accurate answer — and the confidence to ask them back.&lt;/p&gt;

&lt;p&gt;Want to learn more about Chinese traditions, food, and travel spots? Check out &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;more Chinese culture guides&lt;/a&gt; for practical tips written for curious travelers.&lt;/p&gt;

</description>
      <category>beginners</category>
      <category>culture</category>
      <category>learning</category>
    </item>
    <item>
      <title>Redis Stopped Writing at 3 AM: maxmemory + the Eviction Policy Nobody Set</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Sun, 01 Nov 2026 23:29:22 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/redis-stopped-writing-at-3-am-maxmemory-the-eviction-policy-nobody-set-2bcf</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/redis-stopped-writing-at-3-am-maxmemory-the-eviction-policy-nobody-set-2bcf</guid>
      <description>&lt;h1&gt;
  
  
  Redis Stopped Writing at 3 AM: maxmemory + the Eviction Policy Nobody Set
&lt;/h1&gt;

&lt;p&gt;Our cache stopped accepting writes overnight. Not crashed — just refused. Every &lt;code&gt;SET&lt;/code&gt;, &lt;code&gt;HSET&lt;/code&gt;, &lt;code&gt;LPUSH&lt;/code&gt; came back with the same line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;(error) OOM command not allowed when used memory &amp;gt; 'maxmemory'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reads still worked. That's the confusing part, and the part that sends most people down the wrong rabbit hole (restarting, dumping the whole DB, praying). The cache wasn't broken. It was doing exactly what it was told: you gave it a hard ceiling and told it to evict nothing, so the moment it hit the ceiling it stopped instead of growing.&lt;/p&gt;

&lt;p&gt;Here's the 20-minute drill I run before I touch a single config.&lt;/p&gt;

&lt;h2&gt;
  
  
  Minute 1: read the memory snapshot
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;redis-cli INFO memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four lines matter, in order:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Line&lt;/th&gt;
&lt;th&gt;What I'm looking for&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;used_memory_human&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;How close to the ceiling we actually are&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;maxmemory_human&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Whether the ceiling is even set (0 = unlimited = eventual system OOM kill)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;maxmemory_policy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;The real culprit&lt;/strong&gt; — &lt;code&gt;noeviction&lt;/code&gt; means "refuse writes, don't delete anything"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;mem_fragmentation_ratio&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;If it's well over 1.x, the process holds more RSS than logical memory — swap or fragmentation, not a cache-size problem&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The trap I keep finding in production: &lt;code&gt;maxmemory_policy&lt;/code&gt; is &lt;code&gt;volatile-lru&lt;/code&gt; but &lt;strong&gt;not a single key has a TTL&lt;/strong&gt;. &lt;code&gt;volatile-lru&lt;/code&gt; only evicts keys &lt;em&gt;that expire&lt;/em&gt;. With zero expiring keys it behaves exactly like &lt;code&gt;noeviction&lt;/code&gt; — and you get OOM errors despite having a "safe" policy set. This one is invisible unless you check the actual key space.&lt;/p&gt;

&lt;h2&gt;
  
  
  Minute 2: find what's eating it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;redis-cli &lt;span class="nt"&gt;--bigkeys&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That surfaces the largest key per type. Then confirm the suspect:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;redis-cli MEMORY USAGE &amp;lt;key&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two common offenders:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A hash or list that grew to tens of thousands of members because nobody paginated the producer.&lt;/li&gt;
&lt;li&gt;Client output buffers: &lt;code&gt;CLIENT LIST&lt;/code&gt; shows a &lt;code&gt;omem&lt;/code&gt; (output buffer) that's huge — a slow consumer is holding responses in memory instead of the cache holding data.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;When you delete a big key, use &lt;code&gt;UNLINK&lt;/code&gt;, not &lt;code&gt;DEL&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;DEL&lt;/code&gt; is synchronous and blocks the whole instance while it frees — on a multi-hundred-MB hash that can stall the server for seconds. &lt;code&gt;UNLINK&lt;/code&gt; frees it in the background.&lt;/p&gt;

&lt;h2&gt;
  
  
  Minute 3: choose the policy for what you actually are
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pure cache&lt;/strong&gt; → &lt;code&gt;allkeys-lru&lt;/code&gt;. Hot-swap it live:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;  redis-cli CONFIG SET maxmemory-policy allkeys-lru
  redis-cli CONFIG REWRITE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Session store (keys already carry TTLs)&lt;/strong&gt; → &lt;code&gt;volatile-ttl&lt;/code&gt;, evict the soonest-expiring first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Need to protect data&lt;/strong&gt; → &lt;code&gt;noeviction&lt;/code&gt; is fine, &lt;em&gt;if&lt;/em&gt; you alert on memory at 80% and have headroom. &lt;code&gt;noeviction&lt;/code&gt; with no alerting is how you end up here at 3 AM.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The change that would have prevented the page
&lt;/h2&gt;

&lt;p&gt;Set the alert threshold to &lt;strong&gt;80% of &lt;code&gt;maxmemory&lt;/code&gt;&lt;/strong&gt; and make every write path set a TTL by default (&lt;code&gt;SET key value EX 3600&lt;/code&gt; instead of a bare &lt;code&gt;SET&lt;/code&gt;). Then OOM stops being a 3 AM surprise and becomes a normal Monday tuning call.&lt;/p&gt;

&lt;p&gt;Redis memory problems are usually a &lt;strong&gt;key-lifecycle&lt;/strong&gt; problem, not a disk problem: every key should know when it dies the moment it's written.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Tags: redis, caching, linux, devops, troubleshooting&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;More field-tested ops notes and server notes: &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;jiunotes&lt;/a&gt; · &lt;a href="https://ops.jiunotes.top" rel="noopener noreferrer"&gt;ops notes&lt;/a&gt;&lt;/p&gt;

</description>
      <category>database</category>
      <category>debugging</category>
      <category>devops</category>
      <category>redis</category>
    </item>
    <item>
      <title>ECONNRESET at 2 AM: The 5-Minute Drill That Tells You Who Closed the Socket First</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Sat, 31 Oct 2026 09:31:17 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/econnreset-at-2-am-the-5-minute-drill-that-tells-you-who-closed-the-socket-first-3hmc</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/econnreset-at-2-am-the-5-minute-drill-that-tells-you-who-closed-the-socket-first-3hmc</guid>
      <description>&lt;p&gt;Your app logs say &lt;code&gt;ECONNRESET&lt;/code&gt;. Someone closed the socket — but who? Your app, the peer, or a middlebox nobody told you about? Most engineers jump straight to tcpdump, capture 40,000 packets, and drown. The faster path is to read the socket states you already have.&lt;/p&gt;

&lt;p&gt;Here's the 5-minute drill I run, in order.&lt;/p&gt;

&lt;h2&gt;
  
  
  Minute 1: Get the state distribution
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ss &lt;span class="nt"&gt;-tn&lt;/span&gt; | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'{print $1}'&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; | &lt;span class="nb"&gt;uniq&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-rn&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four states actually tell you something:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;What it means&lt;/th&gt;
&lt;th&gt;Threshold that matters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;TIME-WAIT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;You closed first (normal churn)&lt;/td&gt;
&lt;td&gt;Low thousands on a proxy is fine; hundreds of thousands is not&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;CLOSE-WAIT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Peer closed, your app never did&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&amp;gt; 100 is a bug in your code, not the network&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SYN-SENT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Your outbound connect got no answer&lt;/td&gt;
&lt;td&gt;&amp;gt; 100 means a path/firewall problem&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SYN-RECV&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Half-open inbound connections&lt;/td&gt;
&lt;td&gt;Growing = accept queue full or a scan&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The one line I want burned into memory: &lt;strong&gt;CLOSE-WAIT piling up is always a local application bug.&lt;/strong&gt; The peer sent FIN, your process never called &lt;code&gt;close()&lt;/code&gt;, and that socket will sit there forever. Find the owner:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ss &lt;span class="nt"&gt;-tnp&lt;/span&gt; state close-wait
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Minute 2: Check the kernel counters
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;netstat &lt;span class="nt"&gt;-s&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-iE&lt;/span&gt; &lt;span class="s1"&gt;'retrans|reset|listen|overflow'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three counters matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;TCPRetransFail&lt;/code&gt; climbing → packets are being dropped somewhere on the path&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;ListenOverflows&lt;/code&gt; growing → your accept queue is full; real SYNs are being refused&lt;/li&gt;
&lt;li&gt;resets growing steadily → someone is killing connections mid-flight&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If all of them are flat, TCP is not your problem — stop reading socket tables and go look at the application.&lt;/p&gt;

&lt;h2&gt;
  
  
  Minute 3: Attribute the RST
&lt;/h2&gt;

&lt;p&gt;This is the step most people skip, and it's the one that ends the argument about "whose fault" it is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;tcpdump &lt;span class="nt"&gt;-nn&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; any &lt;span class="s1"&gt;'tcp[tcpflags] &amp;amp; tcp-rst != 0'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read the &lt;strong&gt;source IP&lt;/strong&gt; of each RST:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;RST from your own IP&lt;/strong&gt; → your application closed the socket while the peer was still sending. Look for a crash or an over-eager timeout in your own logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Source is the peer&lt;/strong&gt; → they closed first; you sent into a dead socket.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;RST from a third IP (or a load balancer)&lt;/strong&gt; → a middlebox with a short idle timeout is pruning your session. The classic case: your app sends on an idle connection every 90 seconds, but the load balancer's idle timeout is 60 seconds. The LB closed silently; your next packet gets an RST.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That third case is the one I see most, and it's a config mismatch, not a bug: align the LB idle timeout with your app's send interval, or send socket-level keepalives (&lt;code&gt;TCP_KEEPIDLE=60&lt;/code&gt;, &lt;code&gt;TCP_KEEPINTVL=10&lt;/code&gt;, &lt;code&gt;TCP_KEEPCNT=3&lt;/code&gt; — probes start at 60s and give up after ~90s total).&lt;/p&gt;

&lt;h2&gt;
  
  
  One real example
&lt;/h2&gt;

&lt;p&gt;A reverse proxy showed ~4,000 CLOSE-WAIT sockets at 2 AM. &lt;code&gt;ss -tnp state close-wait&lt;/code&gt; showed every one owned by the same worker process. The app had an error path that returned without closing upstream connections — the peer (the upstream) closed, and the leaked sockets sat there forever. The fix was three lines of &lt;code&gt;try/finally&lt;/code&gt;, not a single sysctl. Kernel tuning would have just hidden the leak.&lt;/p&gt;

&lt;h2&gt;
  
  
  The order that works
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;ss -s&lt;/code&gt; — size the problem.&lt;/li&gt;
&lt;li&gt;State distribution (&lt;code&gt;ss -tn | awk ... | uniq -c&lt;/code&gt;) → tells you &lt;strong&gt;which side&lt;/strong&gt; is misbehaving.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;netstat -s&lt;/code&gt; counters → tells you whether the kernel even agrees there's a problem.&lt;/li&gt;
&lt;li&gt;Only then &lt;code&gt;tcpdump&lt;/code&gt;, and only on the specific host/port pair you care about.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Nginx logs tell you the symptom, &lt;code&gt;ss&lt;/code&gt; tells you the state, &lt;code&gt;tcpdump&lt;/code&gt; tells you the packet-level truth — in that order. Tuning &lt;code&gt;sysctl&lt;/code&gt; before you know which side is doing the unusual thing just moves the symptom.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Tags: tcp, networking, linux, debugging, devops&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;More field-tested network and server notes: &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;jiunotes&lt;/a&gt; · &lt;a href="https://ops.jiunotes.top" rel="noopener noreferrer"&gt;ops notes&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>ECONNRESET at 2 AM: The 5-Minute Drill That Tells You Who Closed the Socket First</title>
      <dc:creator>jun liu</dc:creator>
      <pubDate>Fri, 30 Oct 2026 23:21:45 +0000</pubDate>
      <link>https://dev.to/jun_liu_ca219b73a8f1489b5/econnreset-at-2-am-the-5-minute-drill-that-tells-you-who-closed-the-socket-first-46e3</link>
      <guid>https://dev.to/jun_liu_ca219b73a8f1489b5/econnreset-at-2-am-the-5-minute-drill-that-tells-you-who-closed-the-socket-first-46e3</guid>
      <description>&lt;p&gt;Your app logs say &lt;code&gt;ECONNRESET&lt;/code&gt;. Someone closed the socket — but who? Your app, the peer, or a middlebox nobody told you about? Most engineers jump straight to tcpdump, capture 40,000 packets, and drown. The faster path is to read the socket states you already have.&lt;/p&gt;

&lt;p&gt;Here's the 5-minute drill I run, in order.&lt;/p&gt;

&lt;h2&gt;
  
  
  Minute 1: Get the state distribution
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ss &lt;span class="nt"&gt;-tn&lt;/span&gt; | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'{print $1}'&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; | &lt;span class="nb"&gt;uniq&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-rn&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four states actually tell you something:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;What it means&lt;/th&gt;
&lt;th&gt;Threshold that matters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;TIME-WAIT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;You closed first (normal churn)&lt;/td&gt;
&lt;td&gt;Low thousands on a proxy is fine; hundreds of thousands is not&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;CLOSE-WAIT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Peer closed, your app never did&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&amp;gt; 100 is a bug in your code, not the network&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SYN-SENT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Your outbound connect got no answer&lt;/td&gt;
&lt;td&gt;&amp;gt; 100 means a path/firewall problem&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SYN-RECV&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Half-open inbound connections&lt;/td&gt;
&lt;td&gt;Growing = accept queue full or a scan&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The one line I want burned into memory: &lt;strong&gt;CLOSE-WAIT piling up is always a local application bug.&lt;/strong&gt; The peer sent FIN, your process never called &lt;code&gt;close()&lt;/code&gt;, and that socket will sit there forever. Find the owner:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ss &lt;span class="nt"&gt;-tnp&lt;/span&gt; state close-wait
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Minute 2: Check the kernel counters
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;netstat &lt;span class="nt"&gt;-s&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-iE&lt;/span&gt; &lt;span class="s1"&gt;'retrans|reset|listen|overflow'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three counters matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;TCPRetransFail&lt;/code&gt; climbing → packets are being dropped somewhere on the path&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;ListenOverflows&lt;/code&gt; growing → your accept queue is full; real SYNs are being refused&lt;/li&gt;
&lt;li&gt;resets growing steadily → someone is killing connections mid-flight&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If all of them are flat, TCP is not your problem — stop reading socket tables and go look at the application.&lt;/p&gt;

&lt;h2&gt;
  
  
  Minute 3: Attribute the RST
&lt;/h2&gt;

&lt;p&gt;This is the step most people skip, and it's the one that ends the argument about "whose fault" it is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;tcpdump &lt;span class="nt"&gt;-nn&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; any &lt;span class="s1"&gt;'tcp[tcpflags] &amp;amp; tcp-rst != 0'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read the &lt;strong&gt;source IP&lt;/strong&gt; of each RST:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;RST from your own IP&lt;/strong&gt; → your application closed the socket while the peer was still sending. Look for a crash or an over-eager timeout in your own logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Source is the peer&lt;/strong&gt; → they closed first; you sent into a dead socket.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;RST from a third IP (or a load balancer)&lt;/strong&gt; → a middlebox with a short idle timeout is pruning your session. The classic case: your app sends on an idle connection every 90 seconds, but the load balancer's idle timeout is 60 seconds. The LB closed silently; your next packet gets an RST.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That third case is the one I see most, and it's a config mismatch, not a bug: align the LB idle timeout with your app's send interval, or send socket-level keepalives (&lt;code&gt;TCP_KEEPIDLE=60&lt;/code&gt;, &lt;code&gt;TCP_KEEPINTVL=10&lt;/code&gt;, &lt;code&gt;TCP_KEEPCNT=3&lt;/code&gt; — probes start at 60s and give up after ~90s total).&lt;/p&gt;

&lt;h2&gt;
  
  
  One real example
&lt;/h2&gt;

&lt;p&gt;A reverse proxy showed ~4,000 CLOSE-WAIT sockets at 2 AM. &lt;code&gt;ss -tnp state close-wait&lt;/code&gt; showed every one owned by the same worker process. The app had an error path that returned without closing upstream connections — the peer (the upstream) closed, and the leaked sockets sat there forever. The fix was three lines of &lt;code&gt;try/finally&lt;/code&gt;, not a single sysctl. Kernel tuning would have just hidden the leak.&lt;/p&gt;

&lt;h2&gt;
  
  
  The order that works
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;ss -s&lt;/code&gt; — size the problem.&lt;/li&gt;
&lt;li&gt;State distribution (&lt;code&gt;ss -tn | awk ... | uniq -c&lt;/code&gt;) → tells you &lt;strong&gt;which side&lt;/strong&gt; is misbehaving.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;netstat -s&lt;/code&gt; counters → tells you whether the kernel even agrees there's a problem.&lt;/li&gt;
&lt;li&gt;Only then &lt;code&gt;tcpdump&lt;/code&gt;, and only on the specific host/port pair you care about.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Nginx logs tell you the symptom, &lt;code&gt;ss&lt;/code&gt; tells you the state, &lt;code&gt;tcpdump&lt;/code&gt; tells you the packet-level truth — in that order. Tuning &lt;code&gt;sysctl&lt;/code&gt; before you know which side is doing the unusual thing just moves the symptom.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Tags: tcp, networking, linux, debugging, devops&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;More field-tested network and server notes: &lt;a href="https://jiunotes.top/en" rel="noopener noreferrer"&gt;jiunotes&lt;/a&gt; · &lt;a href="https://ops.jiunotes.top" rel="noopener noreferrer"&gt;ops notes&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
