Linux file Descriptor Limits and Socket Leaks: ulimit and /proc/sys/fs/file-max

Originally published on Code Beneath.

You deploy a service, it runs fine for hours, then crashes with "too many open files." The root cause is almost always a file descriptor socket leak, triggered by improper cleanup or hitting the system ulimit. Understanding the two-tier limit system (per-process and system-wide), how to detect leaks, and how to configure both layers properly is essential for any engineer running network services. This article shows you the exact mechanisms, monitoring commands, and working code to diagnose and fix descriptor exhaustion in production.

Understanding Linux File Descriptors and the ulimit System

Every open file, socket, pipe, and device in Linux is represented by a file descriptor (FD), a small integer starting at 0. The first three (0, 1, 2) are reserved for stdin, stdout, and stderr. When you open a TCP socket, bind to a port, accept a connection, or even open a regular file, the kernel assigns the next available FD number.

There are two limits that matter:

1. Per-process soft and hard limits: controlled by ulimit -n and configured in /etc/security/limits.conf.
2. System-wide limit: the total number of file descriptors all processes can hold, stored in /proc/sys/fs/file-max.

If your process hits its soft ulimit, open() returns an error and errno is set to EMFILE (24). If the system runs out of global descriptors, open() fails with ENFILE (23). Both are fatal to a production service. The socket leak occurs when you accumulate unclosed file descriptors until you exhaust one of these limits.

Check Your Current File Descriptor Limits

First, see what your process can do:

# Check soft limit (what the process sees)
ulimit -n

# Check hard limit (maximum you can raise to without root)
ulimit -H -n

# Check system-wide limit
cat /proc/sys/fs/file-max

# See all limits for current shell
ulimit -a

Most distributions ship with 1024 soft limit and 65536 hard limit per process. The system-wide limit is usually 80% of your RAM in KB divided by 10. On a 16 GB machine, expect around 1.3 million descriptors.

Detect File Descriptor Socket Leaks Through Monitoring

A leak is when your process opens descriptors but never closes them. Over time, the count climbs until you hit the ulimit. The easiest way to spot this is to watch the count grow:

# For a running process, count open FDs (replace 12345 with your PID)
ls -1 /proc/12345/fd | wc -l

# Or use lsof
lsof -p 12345 | wc -l

# Watch it grow in real time
watch -n 1 'ls -1 /proc/12345/fd | wc -l'

# See which FDs are open
ls -la /proc/12345/fd/

The /proc filesystem is the most reliable tool here. Each process gets a /proc/[PID]/fd directory with a symlink for every open descriptor. Counting the symlinks tells you exactly how many FDs the process holds right now. This works without installing lsof and is not affected by permission issues that sometimes hide lsof output.

A Runnable Socket Leak Example

Here is a Go program that leaks TCP sockets intentionally, so you can see the symptoms and how to detect them:

package main

import (
    "fmt"
    "net"
    "os"
    "time"
)

func main() {
    if len(os.Args) < 2 {
        fmt.Fprintf(os.Stderr, "usage: %s [leak|clean]\n", os.Args[0])
        os.Exit(1)
    }

    mode := os.Args[1]

    if mode == "leak" {
        // Intentional leak: connect but never close
        fmt.Println("Opening sockets without closing (leak mode)...")
        for i := 0; i < 10000; i++ {
            conn, err := net.Dial("tcp", "127.0.0.1:22")
            if err != nil {
                fmt.Printf("Dial failed at iteration %d: %v\n", i, err)
                break
            }
            // BUG: We never call conn.Close()
            if i%1000 == 0 {
                fmt.Printf("Opened %d connections (leaking)\n", i)
            }
        }
    } else if mode == "clean" {
        // Proper cleanup: close every socket
        fmt.Println("Opening and closing sockets (clean mode)...")
        for i := 0; i < 10000; i++ {
            conn, err := net.Dial("tcp", "127.0.0.1:22")
            if err != nil {
                fmt.Printf("Dial failed at iteration %d: %v\n", i, err)
                break
            }
            conn.Close()
            if i%1000 == 0 {
                fmt.Printf("Opened and closed %d connections\n", i)
            }
        }
    }

    fmt.Println("Sleeping for 60 seconds. Run: watch -n 1 'ls -1 /proc/[PID]/fd | wc -l'")
    time.Sleep(60 * time.Second)
}

Compile and run:

go build -o fd_leak fd_leak.go

# In terminal 1, run leak mode
./fd_leak leak > /tmp/leak.log &
PID=$!

# In terminal 2, watch the FD count climb
watch -n 1 "echo 'FD count:'; ls -1 /proc/$PID/fd 2>/dev/null | wc -l; echo 'Last 5 FDs:'; ls -1 /proc/$PID/fd 2>/dev/null | tail -5"

# Compare with clean mode
./fd_leak clean > /tmp/clean.log &

In leak mode, the FD count will steadily climb to 1024 (the soft limit), then the program will fail with "too many open files." In clean mode, the count stays constant because every socket is closed immediately after use. This is the most obvious diagnostic: a monotonically increasing FD count is a leak.

Proper Resource Cleanup in Production Code

The fundamental rule: always pair open with close, and do it in a way that survives panics or early returns. Here is a robust pattern in Go, Java, and Python:

package main

import (
    "bufio"
    "fmt"
    "io"
    "net"
)

func handleConnection(conn net.Conn) error {
    // Defer ensures Close runs even if we return early or panic
    defer conn.Close()

    reader := bufio.NewReader(conn)
    for {
        line, err := reader.ReadString('\n')
        if err != nil {
            if err == io.EOF {
                return nil // Clean close
            }
            return err
        }
        fmt.Printf("Received: %s", line)
    }
}

func main() {
    listener, err := net.Listen("tcp", ":9999")
    if err != nil {
        panic(err)
    }
    defer listener.Close()

    for {
        conn, err := listener.Accept()
        if err != nil {
            panic(err)
        }
        // Spawn goroutine; handleConnection will close the connection
        go handleConnection(conn)
    }
}

The defer statement is your safety net. Even if handleConnection panics, the defer runs and closes the socket. In Java and Python, use try-finally or context managers:

public void handleConnection(Socket socket) throws IOException {
    try {
        InputStream input = socket.getInputStream();
        BufferedReader reader = new BufferedReader(new InputStreamReader(input));
        String line;
        while ((line = reader.readLine()) != null) {
            System.out.println("Received: " + line);
        }
    } finally {
        socket.close();
    }
}

import socket

def handle_connection(sock):
    try:
        while True:
            data = sock.recv(1024)
            if not data:
                break
            print(f"Received: {data.decode()}")
    finally:
        sock.close()

listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listener.bind(("127.0.0.1", 9999))
listener.listen(5)

while True:
    conn, addr = listener.accept()
    # Spawn thread; handle_connection will close the socket
    import threading
    t = threading.Thread(target=handle_connection, args=(conn,))
    t.daemon = True
    t.start()

Configuring ulimit Permanently

Raising the soft limit to the hard limit for your application is essential. There are three ways to configure ulimit settings:

Method 1: /etc/security/limits.conf

# Add this line to /etc/security/limits.conf
myuser soft nofile 65536
myuser hard nofile 65536

# Or for all users
* soft nofile 65536
* hard nofile 65536

# Restart your shell or log in again for changes to take effect

Method 2: Systemd service file

# In your service file
[Service]
LimitNOFILE=65536
LimitNOFILEHard=65536

Method 3: Raise at startup in your application

Go example using syscall to configure ulimit programmatically:

package main

import (
    "fmt"
    "syscall"
)

func main() {
    // Raise soft limit to hard limit
    var rlim syscall.Rlimit
    err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rlim)
    if err != nil {
        panic(err)
    }

    fmt.Printf("Current: soft=%d, hard=%d\n", rlim.Cur, rlim.Max)

    // Set soft to hard
    rlim.Cur = rlim.Max
    err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rlim)
    if err != nil {
        fmt.Printf("Setrlimit failed: %v (might need CAP_SYS_RESOURCE)\n", err)
    } else {
        fmt.Printf("Raised soft limit to %d\n", rlim.Max)
    }
}

Monitoring System-Wide File Descriptor Limits

Even if a single process can open 65536 descriptors, the system might allow only 2 million total. When you hit this, all processes fail to open files. Monitor both:

# Current system-wide limit
cat /proc/sys/fs/file-max

# Currently allocated descriptors (sum across all processes)
cat /proc/sys/fs/file-nr

# Output format: allocated, unused, max
# Example: 2048    128    262144

If allocated is close to max, you have too many processes holding descriptors. Raise file-max:

# Temporary (lost on reboot)
echo 2097152 | sudo tee /proc/sys/fs/file-max

# Permanent: add to /etc/sysctl.conf
echo "fs.file-max = 2097152" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

End-to-End Monitoring Script

Here is a bash script that monitors both per-process and system-wide limits to catch socket leaks early:

#!/bin/bash
# Monitor file descriptor usage

set -e

if [ $# -eq 0 ]; then
    echo "Usage: $0 [PID]"
    exit 1
fi

PID=$1
INTERVAL=2

echo "Monitoring FD usage for PID $PID (interval ${INTERVAL}s)"
echo "Press Ctrl+C to stop"
echo ""

while true; do
    clear
    echo "=== System-wide ==="
    FILE_NR=$(cat /proc/sys/fs/file-nr)
    FILE_MAX=$(cat /proc/sys/fs/file-max)
    ALLOCATED=$(echo $FILE_NR | awk '{print $1}')
    MAX=$FILE_MAX
    PCT=$((ALLOCATED * 100 / MAX))
    echo "Allocated: $ALLOCATED / $MAX ($PCT%)"
    echo ""

    echo "=== Process $PID ==="
    if [ ! -d "/proc/$PID" ]; then
        echo "Process not found"
        exit 1
    fi

    FD_COUNT=$(ls -1 /proc/$PID/fd 2>/dev/null | wc -l)
    SOFT_LIMIT=$(cat /proc/$PID/limits 2>/dev/null | grep "open files" | awk '{print $4}')
    HARD_LIMIT=$(cat /proc/$PID/limits 2>/dev/null | grep "open files" | awk '{print $5}')

    echo "Open FDs: $FD_COUNT"
    echo "Soft limit: $SOFT_LIMIT"
    echo "Hard limit: $HARD_LIMIT"
    echo ""

    if [ $FD_COUNT -gt 0 ]; then
        echo "Top 10 open FD types:"
        ls -1 /proc/$PID/fd | xargs -I {} sh -c "readlink /proc/$PID/fd/{} 2>/dev/null || echo '(unknown)'" | sort | uniq -c | sort -rn | head -10
    fi

    sleep $INTERVAL
done

Common Mistakes

Mistake	Symptom	Fix
Forgetting to close sockets in a loop	FD count climbs steadily; crashes at ulimit	Use defer, finally, or context managers. Always pair open with close.
Closing in the wrong goroutine/thread	Leaks or double-close panics	Close in the same function that opened. Use defer to guarantee it runs.
Raising only soft limit, not hard limit	Process still crashes after a while	Set both soft and hard, or use Setrlimit at startup.
Not monitoring system-wide limit	All processes fail suddenly even though individual processes look fine	Watch /proc/sys/fs/file-nr and file-max. Raise file-max before you need it.
Checking only lsof, trusting it is complete	Real leak missed because lsof lost access or was slow	Use ls /proc/[PID]/fd as the source of truth. It is instant and never lies.

Integration with Security Best Practices

For production deployment, file descriptor limits are part of a broader hardening strategy. Use ulimit to constrain processes, isolate services in containers with their own limits, and log any failures. A network service should treat EMFILE (too many open files) as a fatal error and alert on-call immediately, because it means either the application has a leak or the infrastructure is under attack (connection flood).

Testing Under Load

Before production, stress-test your service to find leaks early. Use tools like Apache Bench, wrk, or custom load generators to hammer your service for 10 to 60 minutes while watching FD count. If it climbs, you have a leak. If it stays constant, you are ready.

# Example: benchmark while monitoring
PID=$(pgrep -f "your_app")
(watch -n 1 "ls -1 /proc/$PID/fd | wc -l") &
WATCH_PID=$!

# Run load test for 5 minutes
ab -n 100000 -c 100 http://127.0.0.1:8080/

# Stop monitor
kill $WATCH_PID

FAQ

What is the difference between soft and hard limit?

The soft limit is what the process actually enforces. The hard limit is the maximum you can raise the soft limit to without root privilege. A non-root process can lower a hard limit but cannot raise it. Root can raise both arbitrarily.

Can I set per-process limits without modifying limits.conf?

Yes, from inside your application using syscall.Setrlimit (Go), Resource.setRlimit (Java), or resource.setrlimit (Python). This runs at program startup and takes effect immediately. It requires the capability CAP_SYS_RESOURCE (usually available to root or in privileged containers).

Why does my Go server still show EMFILE even after raising ulimit?

Check two things: First, verify the process actually has the raised limit using cat /proc/[PID]/limits. Second, check if you are also hitting the system-wide limit by reading /proc/sys/fs/file-nr. If the allocated count is near file-max, raise file-max.

Is it safe to set file-max very high?

Yes. File descriptors are allocated on demand, so a high file-max does not use memory. On a 64-bit system, setting it to 2 to 4 million is common and safe. Avoid extreme values (10+ million) unless you know the workload.

What if /proc/sys/fs/file-nr shows allocated near max but I do not see high process FD counts?

Some of those descriptors might be held by the kernel itself (inode caches, dentry caches, etc.). Use lsof -p to check individual processes. If totals do not add up, the kernel is likely holding them. This is normal and usually harmless, but if you are running out of descriptors, containers or process isolation helps.

How do I track which FDs are sockets vs files?

Run readlink /proc/[PID]/fd/[FD_NUM]. Socket FDs show as socket:[inode_number]. Regular files show as /path/to/file. Pipes show as pipe:[inode_number]. This helps identify leaks: if you see thousands of socket:12345 entries, you have leaked sockets.

Can I recover from EMFILE without restarting the process?

You can attempt to close old sockets or files, but by the time you hit EMFILE, your process is crippled. The safer approach is to fail fast, alert, and restart. Some frameworks offer circuit breaker patterns that reject new connections before hitting the limit, giving you time to gracefully drain and restart.

Should I use a connection pool to avoid opening too many sockets?

Yes. A connection pool reuses sockets, so you open, say, 50 to 200 persistent connections and keep them alive. This is much more efficient than opening a new socket for every request. See database driver documentation (pgx, mysql, redis) for pool tuning. A well-tuned pool with proper timeouts and health checks will almost never leak.

How does Docker or Kubernetes affect file descriptor limits?

Containers inherit the host kernel's file-max but can have their own per-container limits set via --ulimit (Docker) or resources.limits (Kubernetes). Set both the host-level file-max and the container-level ulimit to ensure consistent behavior. Check limit inside the container with ulimit -n.