DEV Community

jun liu
jun liu

Posted on

EMFILE on Production: A Field Engineer's Guide to File Descriptor Limits

#devops #infrastructure #linux #sre

EMFILE on Production: A Field Engineer's Guide to File Descriptor Limits

It hit at 2 AM. Not with a dramatic stack trace — just a sudden, silent drop in request throughput. Nginx was still serving pages, but new connections weren't being accepted. The error log told the story in one line: accept4() failed (24: Too many open files).

If you've ever seen this on a production server, you know the feeling: the panic of a midnight page, the half-remembered commands, the desperate hunt for which layer of limits is choking your process.

I've seen this error across dozens of servers — Node.js APIs, Python scrapers, Java microservices, Go web servers. Each time, the fix was the same: understand the three layers of Linux file descriptor limits, and tune them correctly. Let me walk you through what EMFILE really means, how to diagnose it fast, and the exact fixes that work in production.

What "File" Really Means in Linux

Here's the first thing most engineers miss: in Linux, a "file" is anything that can be read from or written to. This includes:

  • Regular files (logs, configs, data)
  • Network sockets (TCP connections, HTTP requests)
  • Pipes and FIFOs
  • Device files (/dev/null, TTYs)
  • Inotify watches
  • epoll instances

So when your web server handles 10,000 concurrent connections, it's opening 10,000 "files." When a log rotation script opens old and new log files simultaneously, that's two more. The numbers add up fast.

The kernel assigns each open resource a file descriptor — a small non-negative integer. The first three (0, 1, 2) are reserved for stdin, stdout, and stderr. Everything else starts at 3 and counts up.

By default, most Linux distributions set the per-process soft limit to 1024. For a desktop app, that's plenty. For a server under load? It's a trap waiting to spring.

The Three Layers of Limits

Think of file descriptor limits as a system of nested buckets. All three must be high enough for your workload.

Layer 1: Per-Process Soft Limit

This is what ulimit -n shows you. It's the active ceiling for a single process.

$ ulimit -n
1024
Enter fullscreen mode Exit fullscreen mode

A non-root user can raise this limit on their own, but only up to the hard limit. Once raised, it applies to all child processes spawned from that shell.

The catch: changing this limit in a login shell does nothing for a daemon that started before your change. The limit is copied into the process at fork time. You must restart the service.

Layer 2: Per-Process Hard Limit

This is the ceiling that only root can raise. It's set in /etc/security/limits.conf and applies to new sessions.

$ ulimit -Hn
1024
Enter fullscreen mode Exit fullscreen mode

If soft and hard are equal, no user can raise the limit further without root. The common production setup sets both to 65535.

Layer 3: System-Wide Limit

Even if every process can open 65535 files, the kernel has an overall cap on total file descriptors across all processes. Check it:

$ cat /proc/sys/fs/file-max
823008
Enter fullscreen mode Exit fullscreen mode

This is rarely the bottleneck unless you're running thousands of high-concurrency services on a single box. For most setups, 2 million (2097152) is a safe target.

$ cat /proc/sys/fs/file-nr
18432   0   823008
Enter fullscreen mode Exit fullscreen mode

The three numbers: currently allocated FDs, unused, and the max. If the first number is approaching the third, you need to raise file-max.

Diagnosing EMFILE in Production

When the alert fires, you need answers fast. Here's the diagnostic workflow:

Step 1: Confirm Which Process Is Affected 

# Find the PID of your service
pgrep -f "node|nginx|python|java" | head -5

# Check its current limits
cat /proc/<PID>/limits | grep -A1 "open files"
Enter fullscreen mode Exit fullscreen mode

You'll see output like:

Max open files            1024                1024                files
Enter fullscreen mode Exit fullscreen mode

Both soft and hard are 1024 — that's your smoking gun.

Step 2: Count Current Open File Descriptors 

ls /proc/<PID>/fd | wc -l
Enter fullscreen mode Exit fullscreen mode

If this is near 1024, the process is genuinely at its limit. If it's well below (say, 300), something else is going on — possibly a misconfigured service that's not actually hitting the limit, or a different error masquerading as EMFILE.

Step 3: Identify What's Consuming FDs 

# See what types of files are open
ls -l /proc/<PID>/fd | head -30

# Count by type
ls -l /proc/<PID>/fd | awk '{print $9}' | sed 's|/dev/tcp/||' | sort | uniq -c | sort -rn | head -20
Enter fullscreen mode Exit fullscreen mode

Common patterns:

  • Hundreds of /dev/tcp/* entries → network connections (normal for a busy server)
  • Many log files open → logging misconfiguration
  • Thousands of /tmp/* → temp file leak
  • eventfd, timerfd → modern I/O event loops (epoll/kqueue style)

Step 4: Check for FD Leaks

If the FD count climbs steadily over 30 minutes without corresponding traffic growth, you likely have a leak:

# Snap twice, 30 seconds apart
sleep 30 && ls /proc/<PID>/fd | wc -l
Enter fullscreen mode Exit fullscreen mode

A steady climb = leak. You'll need to identify the code path that opens but doesn't close. Common culprits: unclosed database connections, file handles in loops without finally/defer, socket connections in retry logic.

The Fix: Raising Limits Correctly

Quick Fix: Current Session 

# Raise soft limit to 65535 (must be <= hard limit)
ulimit -n 65535

# Verify
ulimit -n
Enter fullscreen mode Exit fullscreen mode

This works immediately for the current shell and its children. But it resets on logout. Don't use this as your permanent solution — it's a bridge while you configure the proper fix.

Permanent Fix: /etc/security/limits.conf

Edit as root:

sudo nano /etc/security/limits.conf
Enter fullscreen mode Exit fullscreen mode

Add at the end:

* soft nofile 65535
* hard nofile 65535
root soft nofile 65535
root hard nofile 65535
Enter fullscreen mode Exit fullscreen mode

Or, to set per-user limits:

deploy soft nofile 65535
deploy hard nofile 65535
Enter fullscreen mode Exit fullscreen mode

Critical: Users must log out and back in for this to take effect. Service managers reading this file (like systemd with PAM) will pick it up on next start.

The Systemd Trap (Where Most People Get Stuck)

Here's the gotcha that cost me two hours once: systemd ignores /etc/security/limits.conf by default. If your service runs as a systemd unit, the limits you set in limits.conf won't apply.

Instead, create an override:

sudo systemctl edit nginx
Enter fullscreen mode Exit fullscreen mode

This opens an editor with an empty drop-in file. Add:

[Service]
LimitNOFILE=65535
Enter fullscreen mode Exit fullscreen mode

Save and exit, then:

sudo systemctl daemon-reload
sudo systemctl restart nginx
Enter fullscreen mode Exit fullscreen mode

Verify the running process picked it up:

PID=$(pgrep -f nginx | head -1)
cat /proc/$PID/limits | grep "open files"
Enter fullscreen mode Exit fullscreen mode

Expected output:

Max open files            65535               65535               files
Enter fullscreen mode Exit fullscreen mode

You can also preview what systemd will apply without restarting:

systemctl show -p LimitNOFILE nginx
Enter fullscreen mode Exit fullscreen mode

System-Wide Kernel Limit

If you're running many high-FD processes (a VPS with multiple Node.js services, for example), raise the kernel ceiling:

# Check current value
cat /proc/sys/fs/file-max

# Temporarily raise (resets on reboot)
sudo sysctl -w fs.file-max=2097152

# Permanently add to sysctl config
echo "fs.file-max = 2097152" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
Enter fullscreen mode Exit fullscreen mode

2 million FDs system-wide handles ~30 processes at 65535 each with room to spare.

The Verification Mindset

Here's the golden rule: never trust a config file. Trust /proc.

Config files are promises. /proc is reality. After every change:

# 1. Find the running process
PID=$(pgrep -f "your-service")

# 2. Check actual limits
cat /proc/$PID/limits | grep "open files"

# 3. Check actual usage
ls /proc/$PID/fd | wc -l

# 4. Cross-reference
echo "Usage: $(ls /proc/$PID/fd | wc -l) / Limit: $(cat /proc/$PID/limits | grep 'open files' | awk '{print $NF}')"
Enter fullscreen mode Exit fullscreen mode

If usage is climbing toward the limit even after your fix, you haven't solved the problem — you've only delayed the next crash. Go back to Step 4 and hunt for the leak.

Special Cases

Alpine Linux / musl

Alpine doesn't use PAM. Editing /etc/security/limits.conf has zero effect. Instead, set limits in your entrypoint script:

# Dockerfile
RUN apk add --no-cache util-linux
# ...
ENTRYPOINT ["sh", "-c", "ulimit -n 65535 && exec your-app"]
Enter fullscreen mode Exit fullscreen mode

Or use a sidecar/init container that calls prlimit.

macOS / launchd

macOS uses launchd, not systemd. GUI apps and launchd services ignore ulimit. Set limits via a plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>limit.maxfiles</string>
    <key>ProgramArguments</key>
    <array>
        <string>launchctl</string>
        <string>limit</string>
        <string>maxfiles</string>
        <string>65536</string>
        <string>200000</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>
Enter fullscreen mode Exit fullscreen mode

Save to /Library/LaunchDaemons/limit.maxfiles.plist and load it. This trips up almost everyone who develops locally on Mac and deploys to Linux.

Containers and cgroups

Docker containers inherit limits from the host by default. To set custom limits:

docker run --ulimit nofile=65535:65535 your-image
Enter fullscreen mode Exit fullscreen mode

Or in Kubernetes, set it in the pod spec:

securityContext:
  sysctls:
  - name: fs.file-max
    value: "2097152"
Enter fullscreen mode Exit fullscreen mode

Note: cgroups v2 may introduce a pids.max cap. If "too many open files" coincides with fork failures, check the PID cgroup, not just nofile.

Prevention: Monitoring That Actually Works

Raising limits is reactive. Prevention is better. Set up monitoring that catches FD exhaustion before it becomes an outage:

# One-liner for your monitoring agent
watch -n 60 'echo "PID: $(pgrep -f your-service) | Used: $(ls /proc/$(pgrep -f your-service)/fd | wc -l) / Limit: $(cat /proc/$(pgrep -f your-service)/limits | grep open | awk "{print \$(NF-1)}")"'
Enter fullscreen mode Exit fullscreen mode

Or integrate with Prometheus:

# node_exporter collects fd_usage automatically
# Alert when usage > 80% of limit
groups:
  - name: filedescriptors
    rules:
      - alert: HighFileDescriptorUsage
        expr: fd_used / fd_limit > 0.8
        for: 5m
        labels:
          severity: warning
Enter fullscreen mode Exit fullscreen mode

Set alerts at 70% (warning) and 85% (critical). A slow leak gives you days of advance notice; a sudden spike gives you minutes.

The Bottom Line

EMFILE is one of those errors that feels arcane until you understand the three layers. Once you do, it's straightforward:

  1. Diagnose: Check /proc/<PID>/limits and count /proc/<PID>/fd
  2. Fix: Raise limits at the right layer (systemd override, not just limits.conf)
  3. Verify: Trust /proc, not config files
  4. Prevent: Monitor the ratio of used-to-limit before it hits 100%

The next time your phone buzzes at 2 AM with an EMFILE alert, you'll know exactly where to look — and more importantly, you'll know how to explain to your team that this wasn't a mystery, just a configuration that hadn't caught up with the workload.

More ops notes and server reliability guides: jiunotes · ops notes

Top comments (1)

Collapse
 
sgaggjhkjh profile image
Arjun Sharma •

The systemd trap is the detail I would have missed first, because limits.conf looks like the system wide answer until the service ignores it. Your verify step with /proc is what makes this practical rather than a config checklist. Do you also set the warning threshold on the ratio of used to limit, or do you alert only on the absolute count approaching the cap?