Using strace to Debug Linux System Call Errors: Permission Denied and EADDRINUSE

Originally published on Code Beneath.

When a Linux process fails silently or crashes with a cryptic error, your first instinct should be to reach for strace. This system call tracer shows every interaction between your program and the kernel, making it invaluable for diagnosing linux strace permission denied eaddrinuse debugging example scenarios that would otherwise leave you guessing. Rather than installing debug symbols, rebuilding with verbose logging, or squinting at error codes in documentation, strace lets you watch the actual system calls in real time. In this article we trace real failures: a web server unable to bind to port 8080, a file utility hitting permission walls, and a process leaking file descriptors.

What strace Does and Why It Matters

strace intercepts and logs every system call your program makes, along with its arguments and return values. When bind() fails with EADDRINUSE, strace shows you the exact socket state and address. When open() returns permission denied, you see the exact flags and mode bits attempted. This is the lowest level you can observe without kernel source or a debugger, and it requires no application code changes.

The kernel boundary is where most real failures live. A Java application might report "cannot listen on port 8080," but that's a wrapper around a failed bind() syscall. Go's net package might say "address already in use," but the underlying error comes from the kernel returning EADDRINUSE. strace strips the abstraction layer.

Getting Started: Basic strace Usage

Install strace on Debian/Ubuntu with apt-get install strace. On RHEL/CentOS use yum install strace. Then run any command prefixed with strace:

strace -o trace.log ls /tmp

The -o flag writes output to a file instead of stderr. For a running process, attach with:

strace -p 1234 -o trace.log

You will see hundreds of lines. To reduce noise, filter by syscall type with -e:

strace -e openat,open,read,write ls /tmp

The -e flag accepts a comma-separated list of syscall names (no spaces). Some syscalls changed names across kernel versions. On modern Linux, file operations use openat instead of open.

Case Study 1: EADDRINUSE on Port Binding

A Go service binds to port 8080. On restart, it crashes with "address already in use." Let's trace it:

package main

import (
    "fmt"
    "log"
    "net"
)

func main() {
    listener, err := net.Listen("tcp", "127.0.0.1:8080")
    if err != nil {
        log.Fatalf("Listen failed: %v", err)
    }
    defer listener.Close()
    fmt.Println("Listening on port 8080")
    select {}
}

Run the program:

go run server.go &
sleep 1
kill %1
sleep 1
strace -e socket,bind,listen go run server.go

The output shows:

socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
bind(3, {sin_family=AF_INET, sin_port=htons(8080), sin_addr=inet_aton("127.0.0.1")}, 16) = -1 EADDRINUSE (Address already in use)

The socket file descriptor is 3. The bind syscall receives the port in network byte order (htons converts it). The return value -1 and errno EADDRINUSE tell you the port is held by another process.

To find which process owns it:

netstat -tulnp | grep 8080
ss -tulnp | grep 8080

Or trace the entire system with sudo lsof -i :8080. But strace told you exactly which syscall failed and why. If the error had been EACCES (permission denied), you would know the port number is below 1024 and requires root.

To see TCP state transitions, add -e trace=network:

strace -e trace=network,desc go run server.go

This shows socket, bind, listen, accept, shutdown, close, and file descriptor operations all in one trace. The desc group includes dup, dup2, dup3, fcntl, and close, which are critical for finding file descriptor leaks.

Case Study 2: Permission Denied on File Access

A C program tries to read /etc/shadow and fails. Let's trace the error:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void) {
    FILE *f = fopen("/etc/shadow", "r");
    if (!f) {
        perror("fopen");
        return 1;
    }
    char buf[256];
    if (fgets(buf, sizeof(buf), f)) {
        printf("%s\n", buf);
    }
    fclose(f);
    return 0;
}

Compile and trace:

gcc -o shadow shadow.c
strace -e openat ./shadow

Output:

openat(AT_FDCWD, "/etc/shadow", O_RDONLY) = -1 EACCES (Permission denied)

The openat syscall has three arguments: AT_FDCWD means "use the current working directory as the base," the path string, and the flags (O_RDONLY). The return value -1 with errno EACCES tells you the file exists but you lack read permission.

Check the file permissions:

ls -la /etc/shadow

Typically it is -rw-r----- (mode 640) owned by root:shadow. If you run as a regular user, you get EACCES. If you run with sudo, strace shows:

sudo strace -e openat ./shadow

Now openat returns a valid file descriptor (usually 3 or 4). The syscall succeeded. Then read and mmap or read calls follow to pull data into memory.

This is where strace beats guessing. You do not need to check permissions with ls beforehand. strace shows the exact syscall, the exact flags, and the exact errno returned by the kernel. It eliminates the gap between your code's error message and the kernel's decision.

Case Study 3: File Descriptor Leaks

A Python script reads files in a loop and never closes them properly. After many iterations, it crashes with "too many open files." Let's detect the leak:

import os
import sys

def read_file(path):
    f = open(path, 'r')
    data = f.read()
    return data

for i in range(1000):
    try:
        read_file('/etc/hostname')
    except Exception as e:
        print(f"Error at iteration {i}: {e}", file=sys.stderr)
        break

Run with strace filtering for open and close:

strace -e openat,close python3 leak.py 2>&1 | tail -50

You will see openat calls accumulate without matching close calls. The output will show:

openat(AT_FDCWD, "/etc/hostname", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/hostname", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/hostname", O_RDONLY|O_CLOEXEC) = 5
...

The file descriptor numbers increment (3, 4, 5...) but close() is never called. When the limit is reached (usually 1024 by default, tunable with ulimit -n), the next openat fails with EMFILE (too many open files).

Count the ratio with a simple awk script:

strace -e openat,close python3 leak.py 2>&1 | grep -E '^(openat|close)' | awk '{print $1}' | sort | uniq -c

Output:

2 close
  1001 openat

Massive imbalance. The leak is confirmed at the syscall level, before any Python runtime profiler would detect it. Fix it by adding f.close():

def read_file(path):
    f = open(path, 'r')
    data = f.read()
    f.close()
    return data

Re-run strace and you will see close() calls now match openat calls.

Filtering and Output Control

Large traces are overwhelming. Use -e to filter syscalls. Here are common patterns:

Flag	Purpose	Example
-e trace=file	File operations only	openat, readv, write, stat, unlink
-e trace=network	Socket operations only	socket, bind, listen, accept, send, recv
-e trace=process	Process control	fork, exec, exit, wait
-e trace=signal	Signal handlers	sigaction, sigprocmask, kill
-e write	Log syscall arguments and results	All write syscalls
-e read	Log read syscalls	All read syscalls

To see the raw buffer contents of read and write, use -s (string length, default 32):

strace -e write -s 200 cat file.txt

This shows up to 200 bytes of the string being written. Very useful for HTTP requests or log output.

To follow child processes (fork, clone), use -f:

strace -f bash -c 'for i in {1..3}; do (sleep 0.1) & done'

Without -f, you only see the parent bash process. With -f, you see fork, execve, and exit for each child.

To see timing information, use -r (relative time) or -t (absolute time):

strace -r -e socket,bind,listen go run server.go

Each line will show microseconds elapsed since the previous syscall. This reveals performance bottlenecks and synchronization stalls.

Interpreting Errno Values

strace translates errno numbers to names automatically. EADDRINUSE (98), EACCES (13), EMFILE (24), ENOENT (2), EAGAIN (11). But you need to understand what each means:

Errno	Syscall Context	Common Cause
EADDRINUSE	bind	Port already in use or in TIME_WAIT state
EACCES	openat, bind	Insufficient permissions (file mode, port < 1024)
ENOENT	openat, unlink, stat	File or directory does not exist
EMFILE	openat, socket, accept	Process file descriptor limit reached
ECONNREFUSED	connect	No listener on target address:port
EAGAIN	read, write, accept	Resource temporarily unavailable (non-blocking mode)

Reducing Noise with Output Filtering

strace runs your entire program, from startup to exit. Libc initialization alone generates dozens of mmap, mprotect, and brk calls. To focus on your code:

First, run with -c to get a summary of which syscalls were called and how often:

strace -c go run server.go 2>&1 | head -20

This shows you the top syscalls by call count and time spent. Usually you will see poll, read, or write dominating. If bind and listen appear, your server is binding correctly.

Second, skip library initialization by using -e to log only syscalls relevant to your problem:

strace -e trace=file,network -o server.log go run server.go

This drops all signal, process, and descriptor manipulation syscalls and focuses on the actual work.

Third, use grep to search the output file for the failure point:

grep -E '(bind|EADD|EACCES|EMFILE)' server.log

In a 10,000 line trace, this finds the exact line where the failure occurred.

Attaching to Running Processes

You do not have to restart your program. If it is already running and you need to trace it, use -p with the process ID:

strace -p 5432 -e trace=network,desc -o nginx.log

strace will attach and begin logging immediately. Press Ctrl+C to detach without killing the process. This is invaluable for debugging production services without restarting them.

If you want to trace a process only while it is handling a specific request, attach, generate the request in another terminal, then detach. You will see the entire sequence of syscalls triggered by that one request.

Common Mistakes

Mistake 1: Forgetting to redirect stderr. strace writes to stderr by default. If you use > to redirect stdout, strace output mixes with program output. Use 2> file or -o file.

Mistake 2: Not filtering syscalls when the output is huge. Running strace with no filters on a web server generates gigabytes of log files within seconds. Always use -e to narrow scope. Start broad (trace=network,file), then narrow to specific calls (socket,bind,accept).

Mistake 3: Misinterpreting return values. If a syscall returns 0, it succeeded. If it returns -1, it failed and the line will show an errno. If it returns a positive number (for socket or openat), that is a file descriptor. Do not confuse success with failure.

Mistake 4: Running strace without root when tracing system daemons. Some syscalls require elevated privilege to observe. Use sudo strace or run as root if you get "Operation not permitted."

Mistake 5: Assuming the exact order of syscalls in the output. Asynchronous operations, threads, and timer callbacks can reorder syscalls. Always look at the full context: which file descriptor, which address, which flags.

Integration with Security Auditing

strace is an essential security tool for understanding what system calls an untrusted binary actually makes. Rather than reading source code or trusting vendor documentation, you can directly observe every file opened, every socket connected, every signal sent. See our tools guide for strace in combination with seccomp and apparmor profiling.

When sandboxing an application, strace helps you build a minimal seccomp filter. First trace the normal operation to see which syscalls are required. Then write a policy that allows only those syscalls and blocks everything else. This turns permission denied errors into security policies.

FAQ

Can I use strace to debug a segfault?

strace will show syscalls up until the segfault. The last few system calls will reveal what the program was doing before it crashed. For example, if it segfaults while writing to a buffer, you may see a series of read or mmap calls that filled memory. For full debugging use gdb instead: gdb --args ./program args, then run and bt for a backtrace.

How do I trace only failed syscalls?

Use -e signal=none to suppress signal output, then use a shell filter: strace -e trace=network go run server.go 2>&1 | grep -E '= -1'. This shows only syscalls that returned -1 (failure).

Will strace slow down my program significantly?

Yes. Each syscall is intercepted, so tracing adds overhead. For interactive debugging this is fine. For production performance testing, strace is not suitable. Use it to diagnose specific failures, then remove it before measuring performance.

Can I trace a program that reads from stdin?

Yes. Use -i (or pipe input). For example: strace -e read echo hello | ./program. Or run strace interactively without -o, and type input after attaching.

What is the difference between openat and open?

On modern Linux kernels (3.6+), open is automatically converted to openat internally. openat takes an additional directory file descriptor parameter (AT_FDCWD for current directory). strace shows you what the kernel sees, so you will see openat even if your code calls open().

How do I capture the full request and response in a socket trace?

Use -e write,read -s 4096 to log the full buffer contents. For HTTP, this will show the entire GET request and HTTP response headers in the trace. Note that -s is the maximum string length; large responses will be truncated. For very large data, combine with grep to extract the relevant syscalls only.