The Most Interesting Bug I’ve Ever Encountered

The Most Interesting Bug I’ve Ever Encountered

(Or how a library filled up the disk "by magic")

There are annoying bugs, expensive bugs… and then there are those bugs that feel paranormal: you can’t explain what’s happening, the failure mode is not obvious, and the system behaves as if it had a mind of its own.

This is, without a doubt, the most cryptic—and most interesting—bug I’ve ever debugged.

---

The Symptom: Disk space kept vanishing

It started the way these stories often start:

a “normal” process began to fill up the disk.

No clear memory leak. No obvious exception. No suspicious infinite loop in our code.

Just one painful fact:

The file output.log kept growing… until the disk ran out of space.

And the worst part? It wasn’t obvious why.

---

My approach: not analyzing threads… but figuring out what the process was actually doing

The issue was reproducible in a controlled environment, so instead of guessing, I went into debugging mode.

I started taking thread dumps, not because I suspected concurrency issues, but because I needed to answer a simple question:

“What on earth is this process doing right now?”

Thread dumps are often thought of as a tool to inspect thread contention, deadlocks, or runaway parallelism.

But even in a mostly single-threaded process, a thread dump is incredibly useful because it gives you a snapshot of the call stack at that exact moment.

So I took multiple thread dumps over time and looked for patterns.

And that’s when I found the clue.

---

The clue: a suspicious loop in the call stack

The call stack kept showing the process doing something like:

1. Reading a line from a log file
2. Checking a condition
3. Printing the line using System.out.println(...)
4. Repeating

That by itself doesn’t sound terrible… until you realize the log file it was reading was the same file that kept growing.

At this point I knew I wasn’t dealing with a typical bug inside the business logic.

I was dealing with something nastier:

A feedback loop.

---

The Root Cause: System.out had been secretly redirected to a file

Here’s what happened.

Some library code (not ours!) was doing something like:

System.setOut(new PrintStream(new FileOutputStream("output.log")));

In other words, it replaced the global JVM output stream (System.out) and redirected it to a file.

Then later, our process would read that same file:

• If a line matched some condition…
• It would “print it to the console” via System.out.println(line)

But System.out was no longer the console.

It was the log file itself.

So the process was effectively doing this:

1) Read output.log

2) Find a matching line

3) Append the same line back into output.log

4) Continue reading

5) Eventually see that line again (or keep tailing the file)

6) Append again

7) Repeat forever

A self-feeding loop.

A log file eating itself alive.

---

The effect: infinite output amplification

This wasn’t your classic infinite loop like:

while (true) { ... }

It was worse, because the loop emerged from the interaction between two behaviors:

• Redirecting a global output sink
• Reading and re-printing log content

It created an “accidental infinite loop” across system boundaries.

And since disk space is finite, the system didn’t crash—it just kept writing until the disk was completely full.

---

The hardest part: it wasn’t obvious because it wasn’t in our code

This is what made the bug so cryptic:

• System.out is global state for the entire JVM
• The code that changed it lived inside a library
• The process behavior looked legitimate in isolation (“read file → print line”)
• But the hidden coupling created a feedback loop

Once I realized this, the rest was straightforward:

I searched for all occurrences of System.setOut(...) until I found the offending library call.

---

The Fix: restoring System.out as a workaround

Ideally, the library should never have touched System.out in the first place.

But I was told escalating and waiting for the library maintainers to fix it would be “a waste of time,” so I implemented a pragmatic workaround:

PrintStream originalOut = System.out;

try {
  libraryCall();
} finally {
  System.setOut(originalOut);
}

Not pretty, not pure—but it stopped the bleeding.

---

Why this bug stands out

This was one of those rare bugs where:

• the symptoms are severe,
• the cause is deeply hidden,
• and the fix is conceptually simple once you see it.

But seeing it required stepping back and asking:

What is this process actually doing?

Thread dumps were the key—not for analyzing threads, but as a way to observe execution in real time.

---

Lessons learned

1) Libraries should never modify global output streams

Messing with System.setOut() in a shared JVM is playing with fire.

2) Feedback loops can appear from “reasonable” code

Each component was individually defensible:

• “redirect output to a file”
• “read a file and print matching lines”

Together? Disaster.

3) Thread dumps are not just for concurrency

They’re one of the best tools for answering:

“What is my process doing right now?”

---

Final thought

Once I understood what was happening, I couldn’t help but admire the bug.

It was elegant in the worst possible way.

A perfect example of how global state + I/O can create emergent behavior that looks like black magic.

And to this day…

This remains the most interesting bug I’ve ever encountered.