Java ProcessBuilder: Deadlocks, Zombies, and the 64KB Wall

#java #linux #backend #performance

Originally published in Level Up Coding on Medium. You can read the original version here.

Recently at IBM Software Labs, I worked on a task that forced me to understand something many Java developers rarely think about — how Java interacts with the operating system.

Most of our daily work happens safely inside the JVM. Memory management, threads, and file handling — the JVM abstracts these away nicely.

But sometimes you need to step outside. You want to run a shell script, invoke a system binary, or trigger a native tool that no Java library wraps. This is where ProcessBuilder comes in.

ProcessBuilder is the modern Java API for executing native OS commands from Java code. But the moment you call pb.start(), you leave the JVM's safe world. What follows is deadlocks, zombie processes, file descriptor leakage, and race conditions — OS-level problems the JVM cannot protect you from.

ProcessBuilder: The Basics

Before we go any deeper, let me show you what ProcessBuilder actually looks like. At its simplest:

ProcessBuilder pb = new ProcessBuilder("ls", "-la");
// Start it - this is where Java hands control to the OS
Process process = pb.start();
// Read the output
try (BufferedReader reader = new BufferedReader(
        new InputStreamReader(process.getInputStream()))) {
    reader.lines().forEach(System.out::println);
}
// Wait for it to finish and get the exit code
int exitCode = process.waitFor();
System.out.println("Exited with: " + exitCode);

Simple enough. But here is the thing — that single line pb.start() is doing far more than it looks. The moment you call it, you have left the JVM. You are now in the OS's territory; the OS has its own rules.

To understand what those rules are — and why breaking them causes deadlocks, zombies, and resource exhaustion — you need to understand what Linux is actually doing underneath that one method call.

Fundamentals: Life Outside the JVM

Before we talk about Java code, we have to talk about how Linux actually breathes. If you don’t understand the ground your Java process is standing on, the problems we’re about to cover will feel random. They aren’t.

Everything in Linux Is a Process

In Linux, almost everything that does something is a process. Every program, every command, every background service — all of it is a process with a unique ID called a PID.

It all starts with systemd — PID 1. The ancestor of every process on your machine. When you call pb.start() in Java, you aren't just running a command. You are asking the Kernel to birth a new child process, descended from your Java process, descended from systemd.

You can see this lineage yourself:

``shell
pstree -p 1

Look at that tree. systemd(1) at the root. Follow it down — init-systemd → SessionLeader → Relay → bash → jshell → java. Every process has a parent. Every process belongs to a lineage. When you call pb.start(), your new process gets added to this tree as a child of your Java process.

This parent-child relationship isn’t just cosmetic. It’s the mechanism the OS uses to track accountability. The parent is responsible for its children. As we’ll see in the zombie section, when a parent fails to acknowledge a child’s death, the OS has nowhere to put the exit status. The child lingers.

Fork and Exec: How a Process Is Actually Born

So what does the OS actually do when pb.start() is called? It performs a two-step operation every single time.

Fork. The OS creates an exact twin of your Java process. For a split second, two identical Java processes exist in memory. Thanks to a technique called Copy-on-Write, this doesn’t actually double your memory — both processes share the same RAM pages until one of them modifies something.

Exec. The twin then replaces itself entirely. It wipes its own memory, discards the Java bytecode, and loads the binary of the command you passed to ProcessBuilder — bash, ls, whatever it is. The twin is gone. The new process has taken its place.

This is why pb.start() feels instantaneous. You aren't building a new process from scratch — you're cloning and replacing. The OS has been doing this billions of times a day since the 1970s.

/proc: The Kernel’s Brain
There’s a directory on every Linux system called /proc. It looks like a folder. It isn't.

ls /proc

Nothing you see there lives on your hard drive. /proc is a virtual filesystem — a live window the Kernel exposes so you can inspect what's happening inside it in real time. Every numbered directory you see is a running process. Every file inside it is a piece of that process's live state, rendered on demand by the Kernel the moment you read it.

If your Java process has PID 1234, everything about it lives at /proc/1234/ — its memory maps, its open file handles, its current working directory, the exact command that launched it. No special tooling required. The Kernel is just telling you, right there in the filesystem.

We’ll come back to /proc throughout this article. Once you know it's there, you'll never debug a process-related issue the same way again.

File Descriptors: The Ticket Numbers

Linux follows a foundational philosophy — everything is a file. A document on disk, a network socket, a pipe between two processes — the Kernel treats all of it uniformly as a file and hands your process a number to reference it. That number is a File Descriptor, or FD.

Think of FDs as ticket numbers at a Dosa counter. When your process wants to interact with the outside world — read a file, write to a socket, receive input — it hands the Kernel a ticket number. The Kernel knows what that number maps to and routes the operation accordingly.

You can see exactly which FDs your process currently holds:

ls -ll /proc/<PID>/fd

Look at what’s already there before your process does anything meaningful — FD 0, 1, and 2, already open, already pointing somewhere. Those are the Big Three, and every process on Linux starts life with them.

STDIN, STDOUT, STDERR: The Three Doors
Every process starts with three FDs already open:

FD 0 — STDIN. The ear. This is where the process listens for input.

FD 1 — STDOUT. The mouth. This is where the process sends its normal output.

FD 2 — STDERR. The megaphone. This is where the process reports errors.

Now look back at Image 3. Those three FDs — they’re pointing to pipe:[1833473], pipe:[1833474], pipe:[1833475]. That's because this process was spawned by Java using ProcessBuilder. The Kernel has wired those three doors directly to the parent Java process, creating a private channel between them.

When you use ProcessBuilder, your Java code holds the other end of each of these pipes. If you don't manage those connections carefully — drain them, close them, acknowledge them — those pipes back up, those FDs accumulate, and your application hits a wall. That's exactly what the rest of this article is about.