<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Xiao Xiao</title>
    <description>The latest articles on DEV Community by Xiao Xiao (@xiao_xiao_d1a03051e435229).</description>
    <link>https://dev.to/xiao_xiao_d1a03051e435229</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4110904%2Fd114ae22-25fc-431a-ad57-6fd0da903ca0.png</url>
      <title>DEV Community: Xiao Xiao</title>
      <link>https://dev.to/xiao_xiao_d1a03051e435229</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/xiao_xiao_d1a03051e435229"/>
    <language>en</language>
    <item>
      <title>I OOM-killed my production VPS twice with npm ci. Here's the post-mortem.</title>
      <dc:creator>Xiao Xiao</dc:creator>
      <pubDate>Tue, 22 Sep 2026 06:44:09 +0000</pubDate>
      <link>https://dev.to/xiao_xiao_d1a03051e435229/i-oom-killed-my-production-vps-twice-with-npm-ci-heres-the-post-mortem-2gla</link>
      <guid>https://dev.to/xiao_xiao_d1a03051e435229/i-oom-killed-my-production-vps-twice-with-npm-ci-heres-the-post-mortem-2gla</guid>
      <description>&lt;p&gt;I run a small static site — a sleep-cycle calculator I built — on a 2 vCPU / 1612 MB VPS. On September 18th it stopped responding for 10 hours. Two days later it did it again, for 4 hours.&lt;br&gt;
The cause was a single command run on the wrong machine: npm ci.&lt;br&gt;
This is the post-mortem. The interesting part wasn't the fix — it was discovering that two of my three initial guesses were wrong, and that the evidence had been sitting in the logs the whole time. I just didn't know what to look for.&lt;br&gt;
What it looked like from the outside&lt;br&gt;
Cloudflare served 521 and 522 for every request. (521 is "web server is down", 522 is "connection timed out".)&lt;br&gt;
Naturally I blamed Cloudflare, then DNS, then nginx. All three were fine. Spending two days on those assumptions is what made the second outage four times longer than it needed to be.&lt;br&gt;
Here's why 521/522 is a misleading signal: Cloudflare is telling you it can't reach your origin. It has no idea why. Those two numbers describe a symptom from the edge's point of view, and they look identical whether the origin is powered off, firewalled, or alive but unable to answer. Read them as "a Cloudflare problem" and you'll spend your first hour in the wrong place.&lt;br&gt;
The evidence that actually settled it&lt;br&gt;
Two things, and neither of them is a monitoring tool.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Gaps in syslog
/var/log/syslog is written to continuously by dozens of processes. When the whole machine freezes, those writes stop — but the clock doesn't. So when logging resumes, you see a jump in the timestamps:
18:28 → 18:46   18 min
18:47 → 19:10   23 min
19:10 → 19:57   47 min
19:57 → 20:47   50 min
20:47 → 21:48   61 min
21:48 → 01:50  242 min   ← dead
The escalation is the part worth noticing. This isn't a service that died — it's a machine running out of room, each freeze longer than the last.
A gap in syslog is a far stronger signal than "the site was down", because it has essentially one possible cause: nothing could write to disk. "The site was down" has about fifteen.&lt;/li&gt;
&lt;li&gt;The OOM report in kern.log
oom-kill:constraint=CONSTRAINT_NONE, task=npm ci,pid=642435
Out of memory: Killed process 642435 (npm ci)
total-vm:12389432kB, anon-rss:605064kB
Same line three days earlier, different PID. total-vm 12.4 GB, on a machine with 1.6 GB of RAM.
Now the honest caveat, because this number gets misread constantly: total-vm is virtual address space, not resident memory. The process was holding roughly 600 MB of real pages when it was killed. So 12.4 GB is not "this needed 12.4 GB" — it's the number that tells you how aggressively a package manager grabs and touches address space while resolving a dependency tree. The actual killer was memory pressure. Why that pressure was fatal rather than merely slow is the next section.
Why it froze instead of just being slow
Three settings, each defensible on its own, which together removed every escape route:
  Setting
  Value
  Why it hurt
  RAM
  1612 MB
  A handful of small sites, several Node processes, nginx
  Swap
  0
  No buffer at all
  vm.swappiness
  0
  Kernel may only reclaim file cache
swappiness = 0 is the one I'd flag. The advice to set it to zero comes from database tuning — on a box with plenty of RAM, where you'd rather the kernel drop cache than page out a working set. On a 1.6 GB machine with no swap it does something different: the kernel has exactly one tool left, dropping page cache, and it will keep doing that under mounting pressure until it's spending all of its time reclaiming and none of it running anything. That's the freeze. %idle stayed above 90% throughout — the CPU was doing almost nothing, because there was nothing left to run.
If you take one thing from this post: 0 swap plus swappiness 0 is not "swap disabled for performance". It's a machine with no escape hatch.
The fix, in the order that mattered&lt;/li&gt;
&lt;li&gt;A swapfile
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile &amp;amp;&amp;amp; sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab&lt;/li&gt;
&lt;li&gt;Let the kernel actually use it
/etc/sysctl.d/99-memory-guard.conf:
vm.swappiness=10
vm.vfs_cache_pressure=50
10, not 60. The goal isn't a swap-happy box — it's giving the kernel a second option. Swapping a little is enormously better than freezing completely.&lt;/li&gt;
&lt;li&gt;earlyoom, so the next one lasts seconds instead of hours
The core problem was never that npm ci used a lot of memory. It's that the machine spent hours failing to cope before the OOM killer finally acted. earlyoom moves that decision from "hours of thrashing, then the kernel kills something" to "kill the build in seconds".
Drop-in at /etc/systemd/system/earlyoom.service.d/override.conf:
[Service]
ExecStart=
ExecStart=/usr/bin/earlyoom -r 300 -m 8 -s 8 \
--avoid ^(systemd|sshd|nginx|init|dbus-daemon)$ \
--prefer ^(npm|npx|yarn|pnpm|webpack|tsc|vite|node-gyp)$
-m 8 -s 8 means: once memory or swap free drops under 8%, start asking politely, escalating from there. --prefer is the important half — when something has to die, kill the build, not sshd.&lt;/li&gt;
&lt;li&gt;Stop building on the production machine
This is the actual fix. Everything above is containment.
If you genuinely must run it there, put a ceiling on it so it dies alone instead of taking the box with it:
systemd-run --user --scope -p MemoryMax=600M -p MemorySwapMax=1G -- npm ci
The build either fits or gets killed. Either way the rest of the machine keeps answering requests.
The diagnostics I wish I'd run first
# Is there a freeze, and when? Gaps in a log that is normally continuous.
grep -oE '^[0-9-]+T[0-9]{2}:[0-9]{2}' /var/log/syslog | uniq&lt;/li&gt;
&lt;/ol&gt;

&lt;h1&gt;
  
  
  Did the kernel kill something? Two lines per incident.
&lt;/h1&gt;

&lt;p&gt;sudo grep -E "invoked oom-killer|Killed process|Total swap" /var/log/kern.log*&lt;/p&gt;

&lt;h1&gt;
  
  
  Memory history, for the hours nobody was watching.
&lt;/h1&gt;

&lt;p&gt;sar -r -f /var/log/sysstat/sa21 -s 00:00:00 -e 02:30:00&lt;br&gt;
A note on that last one: sysstat had also stopped writing during the longest freeze, so the final sample before the gap timestamped the freeze's start. A monitoring agent that runs on the box you're debugging is, by definition, absent from the worst part of the record.&lt;br&gt;
What I'd do differently&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Read the origin's logs before the CDN dashboard. 521/522 are correct and useless — they say "can't reach origin" and nothing else. The answer was in /var/log/syslog on the far side of the tunnel.&lt;/li&gt;
&lt;li&gt;Treat "no swap" as a deliberate risk decision, not a default. I'd inherited that setting and never re-examined it. Nothing on the box required it.&lt;/li&gt;
&lt;li&gt;Build where memory is cheap, ship the artifact. This site is a static export. The build never needed to touch production at all.&lt;/li&gt;
&lt;li&gt;Alert on a log going quiet, not only on uptime. External checks fire once the site is already down. A missing heartbeat from inside fires at minute five.
If you're seeing this right now
In this order, about five minutes:&lt;/li&gt;
&lt;li&gt;free -m; swapon --show; cat /proc/sys/vm/swappiness — is there pressure, and is there an escape route?&lt;/li&gt;
&lt;li&gt;sudo grep -E "invoked oom-killer|Killed process" /var/log/kern.log* — has the kernel already told you the answer?&lt;/li&gt;
&lt;li&gt;Look for gaps in syslog — full-machine freeze, or one service restarting?&lt;/li&gt;
&lt;li&gt;sar -f /var/log/sysstat/saNN for the hours you weren't looking.&lt;/li&gt;
&lt;li&gt;Then, and only then, go look at the edge.
I wrote all of this down because the second outage existed purely to teach me the first one's lesson. calculatorsleeptime.com is the site that went down — a free sleep-cycle calculator, static export, no backend — but it's the machine underneath it that had the actual problem, and that's the part worth sharing.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>devops</category>
      <category>node</category>
      <category>performance</category>
      <category>showdev</category>
    </item>
  </channel>
</rss>
