DEV Community

Cover image for The Mystery of the Dying 1GB Server
Asep Sayyad
Asep Sayyad

Posted on Originally published at asepsayyad007.in

The Mystery of the Dying 1GB Server

#linux #ubuntu #cloud #devops

How an unsuspected GUI daemon, zero swap, and unrotated Docker logs pushed our cloud node into kernel panics (and how we reclaimed 500MB of free RAM).

The incident report pinged my phone at 02:40 AM.

Our smallest cloud node, a 1-vCPU virtual machine with 1GB of RAM hosting two internal Node.js utilities and a lightweight reverse proxy, stopped responding to HTTP health checks. When I attempted to open an SSH session, the connection hung indefinitely before dropping with a TCP timeout:

ssh: connect to host 198.51.100.42 port 22: Connection timed out
Enter fullscreen mode Exit fullscreen mode

I logged into the cloud provider's web console and opened the out-of-band VNC terminal. The display was frozen on a dump of kernel backtraces. The last four lines on the screen spelled out the entire disaster:

[ 14820.192041] Out of memory: Kill process 1824 (node) score 812 or sacrifice child
[ 14820.194210] Killed process 1824 (node) total-vm:492104kB, anon-rss:312480kB, file-rss:0kB
[ 14820.201844] kernel: oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[ 14820.204512] Kernel panic - not syncing: Out of memory and no killable processes...
Enter fullscreen mode Exit fullscreen mode

The system was dead. The kernel had invoked its Out of Memory (OOM) killer, sacrificed our primary Node.js process, failed to reclaim enough contiguous pages to satisfy outstanding kernel memory allocations, and panicked.

After issuing a hard reboot through the provider API, the machine booted back up. I logged in over SSH and checked memory statistics:

free -h
Enter fullscreen mode Exit fullscreen mode 
               total        used        free      shared  buff/cache   available
Mem:           987Mi       642Mi        60Mi        12Mi       285Mi       285Mi
Swap:             0B          0B          0B
Enter fullscreen mode Exit fullscreen mode

The server had 285 MiB of available memory right after booting. More critically, the swap row read exactly zero.

Running a 1GB cloud node with zero swap space under modern container workloads is like driving down a highway with no brakes. Any brief traffic spike, automated package update, or background memory allocation triggers an instant system collapse.

Here is the exact technical diagnosis and step-by-step remediation we used to turn this fragile node into a rock-solid server with over 530 MiB of free RAM and zero crashes over ninety days of continuous operation.

The Mechanics of Low-Memory Collapse

To understand why a 1GB server collapses so violently, you need to look at how the Linux kernel manages virtual memory.

Linux uses an optimistic memory allocation strategy called memory overcommit. When a process requests memory via malloc() or the brk() and mmap() system calls, the kernel grants virtual address space immediately without allocating physical RAM frames. The kernel assumes that processes will not use all the memory they request at the exact same instant.

Physical memory pages are only mapped when the process actually writes bytes to those virtual addresses, triggering a page fault.

Under normal conditions, physical memory is divided into two primary categories:

  • Anonymous memory: Data belonging to running processes, including the heap, stack, and private mapped memory. This data cannot be discarded because it has no backing file on disk.
  • Page cache: Copies of disk blocks kept in RAM, including executable binaries, shared libraries, and cached file reads.

When applications allocate more memory, the kernel must make room. It does this through page reclamation, coordinated by the background kernel thread kswapd0.

Here is where zero swap becomes fatal.

If a server has a swap device configured, kswapd0 can write cold, inactive anonymous pages to disk, freeing up physical RAM frames for active tasks. But if the server has zero swap, the kernel has only one option: it must evict the page cache.

The kernel begins throwing away cached disk files, shared library segments, and binary code pages. As the page cache shrinks toward zero, every single operation on the machine requires synchronous disk I/O. If the shell needs to run a command, it must fetch the binary from slow virtualized cloud storage block by block.

When the page cache hits rock bottom and anonymous memory fills remaining RAM, kswapd0 spins at 100% CPU in a desperate loop trying to find reclaimable pages. Because there is no swap to write anonymous memory to, the allocator fails.

The kernel invokes out_of_memory(). If the OOM killer selects a process that holds critical locks or cannot release memory fast enough, the entire operating system locks up or panics.

The Culprit You Never Suspect: Headless Desktop Daemons

After rebooting the box, my first priority was auditing where 642 MiB of RAM was going on a fresh boot.

We were running two Docker containers, an Nginx reverse proxy, and basic system utilities. Those workloads should have consumed no more than 250 MiB combined.

I sorted running processes by physical resident memory usage:

ps aux --sort=-%mem | head -n 15
Enter fullscreen mode Exit fullscreen mode

The output delivered an unpleasant surprise:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       912  0.1  4.8 182400 48120 ?        Sl   02:45   0:01 /usr/lib/xorg/Xorg :10 -auth .Xauthority
appuser   1042  0.0  3.9 142100 39180 ?        Sl   02:45   0:00 xfce4-session
appuser   1108  0.0  3.4 118400 34200 ?        S    02:45   0:00 xfwm4
root       845  0.0  2.1  45100 21400 ?        S    02:45   0:00 /usr/sbin/xrdp
root       848  0.0  1.6  38200 16120 ?        S    02:45   0:00 /usr/sbin/xrdp-sesman
appuser   1124  0.0  1.2  89200 12400 ?        S    02:45   0:00 /usr/bin/pulseaudio
message+   612  0.0  0.8  12400  8120 ?        Ss   02:45   0:00 /usr/bin/dbus-daemon --system
Enter fullscreen mode Exit fullscreen mode

A full graphical desktop environment was running quietly in the background on a headless cloud server.

Xorg, xfce4-session, xfwm4, xrdp, and even pulseaudio were initialized and running under systemd.

Why was a graphical desktop running on a production Linux node?

This happens far more often than engineers realize. Many generic hosting providers offer pre-baked OS templates labeled "Ubuntu Desktop" or "Developer Edition" that include XFCE and XRDP out of the box. In other cases, a junior engineer or contractor installs a GUI package like xrdp or xfce4 to troubleshoot a problem visually, leaves the services enabled, and forgets about them.

Let us add up the resident memory footprint of those idle desktop daemons:

  • Xorg: 48.1 MiB
  • xfce4-session: 39.1 MiB
  • xfwm4: 34.2 MiB
  • xrdp and xrdp-sesman: 37.5 MiB
  • pulseaudio: 12.4 MiB
  • Total wasted physical RAM: 171.3 MiB

On a machine with 987 MiB of total memory, 171 MiB represents over 17% of the entire hardware capacity. That memory was locked into physical RAM frames, completely useless, waiting for an RDP login that would never arrive.

Purging the Desktop Stack

If your server does not have a physical monitor attached, it has no business running X11, display managers, or audio daemons.

First, stop and disable the XRDP and display manager services immediately:

sudo systemctl stop xrdp xrdp-sesman lightdm 2>/dev/null || true
sudo systemctl disable xrdp xrdp-sesman lightdm 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

Next, purge the packages completely to reclaim disk space and prevent future package updates from re-enabling them:

sudo apt-get purge -y xrdp xfce4* lightdm pulseaudio x11-common
sudo apt-get autoremove --purge -y
sudo apt-get clean
Enter fullscreen mode Exit fullscreen mode

Finally, tell systemd to boot strictly into the multi-user console target rather than the graphical target:

sudo systemctl set-default multi-user.target
Enter fullscreen mode Exit fullscreen mode

Checking memory immediately afterward showed our available RAM jump from 285 MiB to 448 MiB.

We had recovered over 160 MiB of clean memory in two minutes, but the server was still missing its most critical safety feature: swap.

Building a 2GB Emergency Swap Architecture

There is a widespread misconception across cloud forums that modern Linux servers running on SSDs or NVMe drives should never use swap. People claim that swap causes SSD wear or introduces unacceptable latency.

That advice is completely wrong for low-memory cloud instances.

Swap is not meant to serve as cheap, slow RAM for active computing. If your active application working set exceeds physical RAM and constantly reads from swap, your performance will suffer regardless of the disk type.

The true purpose of swap is twofold:

  1. An emergency escape valve: It gives the kernel a buffer to absorb memory spikes (such as an npm install, a database migration, or a sudden burst of HTTP requests) without triggering the OOM killer.
  2. A parking lot for dormant memory: Operating system daemons, initialization code, and idle processes often allocate memory during startup that they never touch again. Moving those cold pages to swap frees up expensive physical RAM for your active application heap and filesystem cache.

On a 1GB cloud VPS, running without swap is negligence. We need a 2GB swapfile.

Creating the Swapfile Safely

You can allocate a swapfile using fallocate or dd. While fallocate is faster because it manipulates filesystem metadata directly, some virtualized filesystems or copy-on-write systems (like Btrfs) can create fragmented files or sparse allocation holes that the kernel refuses to swap to.

To ensure contiguous, reliable allocation on any cloud block storage, use dd:

sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress
Enter fullscreen mode Exit fullscreen mode

This writes exactly 2,048 megabytes of zeroed blocks to /swapfile.

Locking Down Permissions

A swapfile contains raw dumps of kernel and process memory. That memory contains database credentials, API tokens, session cookies, and private encryption keys. If regular users can read this file, your security model is destroyed.

Set strict read-and-write permissions for the root user only:

sudo chmod 600 /swapfile
Enter fullscreen mode Exit fullscreen mode

Verify that the permissions show -rw-------:

ls -l /swapfile
Enter fullscreen mode Exit fullscreen mode

Formatting and Enabling Swap

Format the file with the Linux swap header structure:

sudo mkswap /swapfile
Enter fullscreen mode Exit fullscreen mode

The command will output the swap space version and its unique UUID:

Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
no label, UUID=a4891b6f-7b89-4e89-a212-098276f512c0
Enter fullscreen mode Exit fullscreen mode

Now, activate the swapfile:

sudo swapon /swapfile
Enter fullscreen mode Exit fullscreen mode

Verify that the kernel has recognized the new swap capacity:

swapon --show
Enter fullscreen mode Exit fullscreen mode 
NAME      TYPE  SIZE USED PRIO
/swapfile file    2G   0B   -2
Enter fullscreen mode Exit fullscreen mode

Ensuring Persistence Across Reboots

If the server reboots, /swapfile will not activate automatically unless it is registered in /etc/fstab.

Add the entry to /etc/fstab:

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
Enter fullscreen mode Exit fullscreen mode

Verify that /etc/fstab is syntactically correct and mounts cleanly:

sudo mount -a
Enter fullscreen mode Exit fullscreen mode

Our low-memory instance now had a 2GB safety cushion. But leaving swap with default kernel parameters introduces another dangerous failure mode: disk thrashing.

Taming Kernel Swappiness and Cache Pressure

By default, the Linux kernel ships with vm.swappiness = 60.

Swappiness is a kernel parameter that controls the balance between reclaiming page cache (file-backed memory) and swapping out anonymous memory (application data). The value ranges from 0 to 200 on modern kernels (historically 0 to 100).

When vm.swappiness = 60, the kernel is relatively aggressive about pushing anonymous memory to swap even when there is plenty of physical RAM available.

On a desktop workstation or a database server with 64GB of RAM, this behavior helps keep the filesystem cache large and responsive. But on a 1GB cloud VPS running on budget shared block storage, an aggressive swappiness value is disastrous.

When a background task triggers minor disk I/O, the kernel eagerly moves active Node.js or Python runtime pages into swap. The next time an incoming HTTP request hits your web app, the process stalls while the kernel reads its pages back from virtualized disk into RAM.

Your API response times spike from 25 milliseconds to 1,800 milliseconds.

We do not want the kernel swapping application code during routine operations. We want swap utilized strictly as a safety reserve when physical memory is genuinely exhausted.

To achieve this, we set vm.swappiness = 10.

Tuning VFS Cache Pressure

Another critical sysctl parameter for low-memory systems is vm.vfs_cache_pressure.

This parameter controls the kernel's tendency to reclaim directory entries (dentries) and inode objects from memory compared to page cache and swap. The default value is 100.

On cloud instances with limited I/O operations per second (IOPS), constantly dropping directory and inode caches forces the operating system to re-read file metadata from slow storage over and over.

Lowering vm.vfs_cache_pressure to 50 tells the kernel to retain directory structures and inode metadata in memory longer, speeding up filesystem lookups and lowering CPU wait times.

Applying Persistent Sysctl Rules

Create a dedicated configuration file inside /etc/sysctl.d/:

sudo nano /etc/sysctl.d/99-swappiness.conf
Enter fullscreen mode Exit fullscreen mode

Insert the following parameters:

# Reduce aggressive swapping to preserve latency on small cloud nodes
vm.swappiness = 10

# Retain directory and inode cache longer to prevent disk read stalls
vm.vfs_cache_pressure = 50
Enter fullscreen mode Exit fullscreen mode

Apply the new parameters immediately without rebooting:

sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

Verify the active values:

cat /proc/sys/vm/swappiness
cat /proc/sys/vm/vfs_cache_pressure
Enter fullscreen mode Exit fullscreen mode

With vm.swappiness = 10, the kernel keeps your active processes pinned to fast physical RAM. It will only touch the 2GB swap partition when available RAM drops below 10-15%, preventing premature disk thrashing while still protecting the node from OOM panics.

The Silent Memory Killer: Unbounded Docker Logs

While inspecting our second low-memory VPS, I noticed a baffling symptom: physical RAM was stable, but swap usage had bloated to 538 MiB.

What was consuming hundreds of megabytes of swap on a system running just three small Docker containers?

The answer was hiding inside /var/lib/docker/containers/.

By default, Docker uses the json-file logging driver. Unless you explicitly configure log rotation flags, Docker has no maximum file size limit. Every console.log(), access record, and error stack trace output by your containers is appended to a JSON file on disk forever:

/var/lib/docker/containers/<container-id>/<container-id>-json.log
Enter fullscreen mode Exit fullscreen mode

Let us check the sizes of those log files on disk:

sudo du -sh /var/lib/docker/containers/*/*-json.log
Enter fullscreen mode Exit fullscreen mode

On our secondary node, a single container had generated an 8.4GB log file over four months:

8.4G    /var/lib/docker/containers/a7f10b28.../a7f10b28...-json.log
Enter fullscreen mode Exit fullscreen mode

How does an 8GB log file on disk consume memory and swap?

First, whenever a monitoring agent, log shipper, or developer runs docker logs --tail 100 <container>, the Docker daemon attempts to read and parse the giant JSON file.

Second, the Linux kernel allocates page cache buffers to service these massive file reads. As these buffers expand, the kernel feels memory pressure and pushes dormant container memory into swap.

Third, if the disk space fills up completely, system daemons fail to write temporary files, SQLite databases lock up, and memory allocations fail across the board.

Configuring Global Log Rotation in Docker

You do not need to configure log limits manually inside every single docker-compose.yml file. You can enforce global rotation policies across all containers at the daemon level.

Edit or create /etc/docker/daemon.json:

sudo nano /etc/docker/daemon.json
Enter fullscreen mode Exit fullscreen mode

Add the following configuration:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
Enter fullscreen mode Exit fullscreen mode

This configuration tells Docker that no container log file may exceed 10 megabytes. Docker will keep a maximum of 3 rotated files (.log, .log.1, .log.2), capping total log storage per container at 30 megabytes.

Reload the Docker daemon configuration safely without stopping active containers:

sudo systemctl reload docker
Enter fullscreen mode Exit fullscreen mode

Safely Truncating Existing Bloated Logs

Reloading Docker enforces the 10MB limit on newly rotated files, but it does not shrink existing multi-gigabyte log files already open on disk.

Do not delete the .log files with rm. If a running container has an open file descriptor pointing to that file, deleting it leaves the file descriptor open in an unlinked state. The disk space will not be freed until the container stops.

Instead, truncate the files in place:

sudo sh -c 'for f in /var/lib/docker/containers/*/*-json.log; do truncate -s 0 "$f"; done'
Enter fullscreen mode Exit fullscreen mode

The truncate -s 0 command zeros out the contents instantly while keeping the existing file descriptor valid.

Within seconds, gigabytes of disk space were freed, kernel buffer pressure dissipated, and swap usage plummeted.

Enforcing Container Memory Limits and Protecting SSH

If you run Docker containers on a 1GB host without memory constraints, a single runaway process inside a container can consume every byte of RAM and crash the underlying host.

Every container running on a low-memory VPS must have explicit resource limits defined in its docker-compose.yml file.

Here is a hardened configuration for our Node.js service:

version: "3.8"

services:
  web-app:
    image: node:20-alpine
    restart: unless-stopped
    command: node server.js
    environment:
      - NODE_ENV=production
      # Limit V8 heap allocation to 192MB to avoid OOM crashes
      - NODE_OPTIONS=--max-old-space-size=192
    deploy:
      resources:
        limits:
          cpus: "0.75"
          memory: 256M
        reservations:
          memory: 128M
Enter fullscreen mode Exit fullscreen mode

Notice the combination of --max-old-space-size=192 and memory: 256M.

If you only set the Docker memory limit without configuring Node.js, the V8 garbage collector will assume it has access to the full host memory. V8 will postpone garbage collection until the container hits the 256MB threshold, causing Docker to kill the container.

By setting max-old-space-size below the container limit, V8 aggressively runs garbage collection when heap usage reaches 192MB, keeping the process well within its 256MB boundary.

Protecting SSH from the OOM Killer

If an unexpected memory spike occurs despite your limits, you still need to access the server over SSH to troubleshoot.

By default, the Linux OOM killer can choose sshd as a victim if it needs to free memory. If sshd is terminated, you are locked out of your server and forced to perform a hard reboot.

You can protect the SSH daemon by adjusting its OOM score modifier: /proc/<pid>/oom_score_adj.

The score ranges from -1000 (never kill) to +1000 (always kill first).

Create a systemd service override for SSH:

sudo mkdir -p /etc/systemd/system/ssh.service.d/
sudo nano /etc/systemd/system/ssh.service.d/override.conf
Enter fullscreen mode Exit fullscreen mode

Add the following configuration:

[Service]
OOMScoreAdjust=-500
Enter fullscreen mode Exit fullscreen mode

Reload systemd and restart the SSH daemon:

sudo systemctl daemon-reload
sudo systemctl restart ssh
Enter fullscreen mode Exit fullscreen mode

Verify that the active sshd process has the updated score:

cat /proc/$(pgrep -f "/usr/sbin/sshd" | head -n 1)/oom_score_adj
Enter fullscreen mode Exit fullscreen mode

The output will confirm -500.

Even if your web application consumes memory and triggers the OOM killer, the kernel will sacrifice other processes before touching SSH, allowing you to log in and diagnose the issue cleanly.

The Results: Before vs. After Telemetry

After applying these four optimizations, the change in server stability and memory availability was immediate and dramatic.

Here is the operational comparison on our primary 1GB cloud node:

Initial State (Vulnerable to OOM):
Physical RAM Total: 987 MiB
Available Memory: 285 MiB (28.8% free)
Swap Space Total: 0 B
Swap Space Used: 0 B
Idle Background Load: 0.45
OOM Outages: 3 panics in 48 hours
Rogue Daemons: Xorg, XFCE4, XRDP consuming 171 MiB

Final Hardened State:
Physical RAM Total: 987 MiB
Available Memory: 531 MiB (53.8% free)
Swap Space Total: 2,048 MiB (2.0 GiB)
Swap Space Used: 18 MiB (dormant pages only)
Idle Background Load: 0.04
OOM Outages: 0 panics over 90+ consecutive days
Rogue Daemons: 0 (pure headless multi-user console)

Available memory jumped from 285 MiB to 531 MiB. We recovered 246 MiB of clean physical RAM, nearly doubling our available memory headroom on the exact same budget cloud hardware.

An Interesting Fact About the Linux OOM Killer

The Linux kernel does not choose its victims randomly. When memory exhaustion triggers the OOM killer, the kernel evaluates every runnable process using the badness() function located in mm/oom_kill.c.

The algorithm calculates an integer score between 0 and 1000 for each candidate process:

points = (process_rss + page_table_pages + swap_pages) * 1000 / total_ram_pages
Enter fullscreen mode Exit fullscreen mode

The process using the highest percentage of memory receives the highest baseline score.

However, the kernel adds a protective heuristic for privileged tasks: if a process runs as root (has_capability_noaudit(p, CAP_SYS_ADMIN)), the algorithm subtracts 30 points from its score. The kernel gives system daemons a slight advantage over regular user processes.

Finally, the kernel adds the value of /proc/<pid>/oom_score_adj directly to the calculated points.

If a process has an oom_score_adj of -1000, the calculation returns 0 immediately, granting that process absolute immunity from the OOM killer. If you set it to +1000, that process becomes the immediate sacrifice the moment memory runs dry.

The 5-Minute Hardening Checklist for 1GB VPS Instances

Whenever you provision a new low-spec cloud VPS, run through this five-minute checklist before deploying your application workloads:

  1. Check for graphical bloat: Run ps aux --sort=-%mem | head -n 10. If you see xorg, xrdp, or xfce, purge them immediately and switch to multi-user.target.
  2. Provision a 2GB swapfile: Allocate with dd, set permissions to chmod 600, format with mkswap, and persist in /etc/fstab.
  3. Set swappiness to 10: Create /etc/sysctl.d/99-swappiness.conf with vm.swappiness = 10 and vm.vfs_cache_pressure = 50.
  4. Enforce Docker log rotation: Configure max-size: 10m and max-file: 3 in /etc/docker/daemon.json and reload the daemon.
  5. Protect SSH: Set OOMScoreAdjust=-500 in a systemd override for ssh.service so you never get locked out during an incident.

Have you ever logged into a budget cloud VPS only to discover a rogue desktop environment or unrotated Docker logs consuming half your memory? What is the strangest daemon you have discovered running on a production node?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

Asep Sayyad is a Linux and DevOps engineer passionate about Linux administration, automation, cloud technologies, containers, and open-source software. He enjoys solving real-world infrastructure challenges and sharing practical knowledge through in-depth technical articles, tutorials, and hands-on guides.

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

  • Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
  • Sharing this article with fellow Linux and DevOps engineers.
  • Connecting on LinkedIn and GitHub to discuss cloud infrastructure and Linux internals.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

© 2026 Asep Sayyad

Top comments (0)