DEV Community: Miguel Novelo

The Most Interesting Bug I’ve Ever Encountered

Miguel Novelo — Fri, 16 Jan 2026 03:03:01 +0000

The Most Interesting Bug I’ve Ever Encountered

(Or how a library filled up the disk "by magic")

There are annoying bugs, expensive bugs… and then there are those bugs that feel paranormal: you can’t explain what’s happening, the failure mode is not obvious, and the system behaves as if it had a mind of its own.

This is, without a doubt, the most cryptic—and most interesting—bug I’ve ever debugged.

The Symptom: Disk space kept vanishing

It started the way these stories often start:

a “normal” process began to fill up the disk.

No clear memory leak. No obvious exception. No suspicious infinite loop in our code.

Just one painful fact:

The file output.log kept growing… until the disk ran out of space.

And the worst part? It wasn’t obvious why.

My approach: not analyzing threads… but figuring out what the process was actually doing

The issue was reproducible in a controlled environment, so instead of guessing, I went into debugging mode.

I started taking thread dumps, not because I suspected concurrency issues, but because I needed to answer a simple question:

“What on earth is this process doing right now?”

Thread dumps are often thought of as a tool to inspect thread contention, deadlocks, or runaway parallelism.

But even in a mostly single-threaded process, a thread dump is incredibly useful because it gives you a snapshot of the call stack at that exact moment.

So I took multiple thread dumps over time and looked for patterns.

And that’s when I found the clue.

The clue: a suspicious loop in the call stack

The call stack kept showing the process doing something like:

Reading a line from a log file
Checking a condition
Printing the line using System.out.println(...)
Repeating

That by itself doesn’t sound terrible… until you realize the log file it was reading was the same file that kept growing.

At this point I knew I wasn’t dealing with a typical bug inside the business logic.

I was dealing with something nastier:

A feedback loop.

The Root Cause: `System.out` had been secretly redirected to a file

Here’s what happened.

Some library code (not ours!) was doing something like:

System.setOut(new PrintStream(new FileOutputStream("output.log")));

In other words, it replaced the global JVM output stream (System.out) and redirected it to a file.

Then later, our process would read that same file:

If a line matched some condition…
It would “print it to the console” via System.out.println(line)

But System.out was no longer the console.

It was the log file itself.

So the process was effectively doing this:

1) Read output.log

2) Find a matching line

3) Append the same line back into output.log

4) Continue reading

5) Eventually see that line again (or keep tailing the file)

6) Append again

7) Repeat forever

A self-feeding loop.

A log file eating itself alive.

The effect: infinite output amplification

This wasn’t your classic infinite loop like:

while (true) { ... }

It was worse, because the loop emerged from the interaction between two behaviors:

Redirecting a global output sink
Reading and re-printing log content

It created an “accidental infinite loop” across system boundaries.

And since disk space is finite, the system didn’t crash—it just kept writing until the disk was completely full.

The hardest part: it wasn’t obvious because it wasn’t in our code

This is what made the bug so cryptic:

System.out is global state for the entire JVM
The code that changed it lived inside a library
The process behavior looked legitimate in isolation (“read file → print line”)
But the hidden coupling created a feedback loop

Once I realized this, the rest was straightforward:

I searched for all occurrences of System.setOut(...) until I found the offending library call.

The Fix: restoring `System.out` as a workaround

Ideally, the library should never have touched System.out in the first place.

But I was told escalating and waiting for the library maintainers to fix it would be “a waste of time,” so I implemented a pragmatic workaround:

PrintStream originalOut = System.out;

try {
  libraryCall();
} finally {
  System.setOut(originalOut);
}

Not pretty, not pure—but it stopped the bleeding.

Why this bug stands out

This was one of those rare bugs where:

the symptoms are severe,
the cause is deeply hidden,
and the fix is conceptually simple once you see it.

But seeing it required stepping back and asking:

What is this process actually doing?

Thread dumps were the key—not for analyzing threads, but as a way to observe execution in real time.

Lessons learned

1) Libraries should never modify global output streams

Messing with System.setOut() in a shared JVM is playing with fire.

2) Feedback loops can appear from “reasonable” code

Each component was individually defensible:

“redirect output to a file”
“read a file and print matching lines”

Together? Disaster.

3) Thread dumps are not just for concurrency

They’re one of the best tools for answering:

“What is my process doing right now?”

Final thought

Once I understood what was happening, I couldn’t help but admire the bug.

It was elegant in the worst possible way.

A perfect example of how global state + I/O can create emergent behavior that looks like black magic.

And to this day…

This remains the most interesting bug I’ve ever encountered.

Improving Code Clarity with Functional Programming in Java.

Miguel Novelo — Thu, 28 Sep 2023 01:13:13 +0000

Today, after a month of working on a new feature, we were ready to have a bug bash, which essentially saved me before ramping this new feature to the public.

As I began debugging my code, I found myself examining a piece of code that utilizes streams, a crucial tool for functional programming in Java when working with collections.

I appreciate how elegant, concise, and readable the code becomes when using map to transform items, apply filters, find minimum and maximum values. However, this time, my lack of experience using these features worked against me.

So, What happened?

I was working with a piece of code performing some transformations like this:

entities.stream()
  .map(Entity::getId)
  .map(new EntityCriteria()::setEntity)
  .collect(Collectors.toList())

If you're more experienced than I was when I wrote this code, you might have already spotted the issue. Allow me to explain:

entities is a collection (List) and its elements are type of Entity
map(Entity::getId) will transform the elements into their IDs
The intention of map(new EntityCriteria()::setEntity) was to transform the entity IDs into EntityCriteria objects with the entityId. However, this happened, but not in the right way.

Understanding the Concepts.

To comprehend what is wrong with number 3, we need to grasp some functional programming concepts:

Entity::getId from the Entity class uses the :: operator to retrieve a Function called getId. It does NOT retrieve the actual ID.
This Function will be applied to the Entity elements to retrieve their IDs because Entity is a class, and getId doesn't accept any arguments.
When we call map(new EntityCriteria()::setEntity), new EntityCriteria() creates a new object. Then ::setEntity on that object generates a Function capable of applying setEntity to the new EntityCriteria object.

Root Cause

The Function created by new EntityCriteria()::setEntity is always applied to the same EntityCriteria object, It doesn't create new objects and then retrieve the function to apply setEntity.

The Most Important Question: Why this wasn't caught in Unit Tests ?

The mocked data used in the unit tests, replicated the same buggy logic for transforming the entities, so everything appeared to work.

Possible solutions:

Using Lambdas

map(entityId -> new EntityCriteria().setEntity(entityId))

When we use lambdas, we define the function where the entityId on the left is the input to the function, and the right part is the body of the function (In this case, the transformation logic).

Creating a new method to be applied in `map`

private static EntityCriteria createEntityCriteria(long entityId) {
  return new EntityCriteria().setEntity(entityId);
}

...

entities.stream()
  .map(Entity::getId)
  .map(MyClass::createEntityCriteria)
  .collect(Collectors.toList())

This approach creates a Function that applies createEntityCriteria and that function generates a new EntityCriteria object for each element.

Conclusion

Everybody makes mistakes, so make sure to thoroughly test your code.
When writing mock data in unit tests, avoid replicating the logic for building the transformation if that is what you are actually testing.

Setting probability to choose something randomly

Miguel Novelo — Tue, 02 Nov 2021 21:52:22 +0000

Playing at Tetris I always wonder how some shapes are most common that they appear than others. Basically, you can expect that you almost never see the line, probably you'll see twice as often the square, then the L shapes and the kind-of zigzag shapes are the most common shapes.

If you ever think about, how would you implement the shape choosing, just by using the pseudo-random to select the shape, the probability to get any shape will be the same. But we can think some other approaches to change the probability.

Let's say we have the following shapes with their chances to be picked:

class Shape:
  def __init__(self, name, chances):
    self.name = name
    self.chances = chances

shapes = [
    Shape("ZigZag", 8),
    Shape("LeftZigZag", 8),
    Shape("L", 4),
    Shape("LeftL", 4),
    Shape("Square", 2),
    Shape("Line", 1),
]

There are 6 shapes, and the total chances (sum of all chances) are: 27. So the chances of getting a ZigZag will be 8 out of 27, when the chances of getting a line would be 1 out of 27.

Approach 1 Creating a list with all the chances

The first approach that came to my mind is creating a list of size 27 and fill it with all the shapes, repeating the shape as many chances it has to appear. This way we have the same chance to pick any of the 27 indexes of the list, but if a shape has 8 slots, this shape will have 8 out of 27 chances to appear:

class RandomShapePicker:
  def __init__(self, shapes):
    self.chances = []
    for shape in shapes:
      for _ in range(shape.chances):
        self.chances.append(shape)

  def pick_shape(self):
    return random.choice(self.chances)

Analyzing the performance, to create the RandomShapePicker, memory will be in function of the total sum of chances (In this case 27) and the time complexity will be in function of the sum of chances as well: O(C), where C is the total sum of chances.

Picking a shape time complexity will be a constant as we get a random index and access directly to that index. We don't need extra space either which is constant as well: O(1).

This works very well for a small list which can potentially call pick_shape multiple times as we can expect constant time for these calls.

The problem with this approach is when the number of choices is large, and we manage large number of chances. Imagine that the sum of chances is 1000000 which means that our list should have 1M spots. Even if the different choices are low (Let's say 1000).

For this particular problem would be more efficient not having multiple spots for the same choice. But then, we have to think how we are going to manage that a Line shape has 1/27 chances to appear.

Approach 2 Creating a list with the cumulative sum of chances.

Rethinking our solution:

Picking a number between 0 and 26 (27 chances) the chances of getting:

a number between 0 and 7 is 8/27 (ZigZag have 8 out of 27 chances)
a number between 8 and 15 is 8/27 (LeftZigZag have 8 out of 27 chances)
a number between 16 and 19 is 4/27 (L have 4 out of 27 chances)
a number between 20 and 23 is 4/27 (LeftL have 4 out of 27 chances)
a number between 24 and 25 is 2/27 (Square have 2 out of 27 chances)
The number 26 is 1/27 (Line have 1 out of 27 chances)

There is a pattern with the number of chances and the cumulative sum. If we store this cumulative sum in a list, we will have:

self.chances = [8, 16, 20, 24, 26, 27]

We can use binary search to find the insertion point for that random number, and then select the shape based on the chances that we have to get that shape:

For example, if we get number 21, which is in the range of 20 and 23 (4 chances to get a LeftL) The insertion index for that 21 is
3 (The one with the value 24) which is the same index as the LeftL shape.

The implementation would be:

class RandomShapePicker:
  def __init__(self, shapes):
    self.shapes = shapes
    self.chances = []
    self.total_chances = 0
    for shape in shapes:
      self.total_chances += shape.chances
      self.chances.append(self.total_chances)

  def pick_shape(self):
    choice = random.random() * self.total_chances
    i = bisect_left(self.chances, choice)
    return self.shapes[i]

This implementation will trade off time complexity from pick_shape to improve the time and space complexity for the RandomShapePicker creation.

The constructor is now O(N) time and space complexity (N is the number of choices which is less than number of chances), and pick_shape time complexity is now O(logN) which is still a decent performance for multiple callings of pick_shape.

Conclusion

We need to think about trade-offs here. Probably is not a bad idea that we are calling a big number of times the pick_shape() or pick_option() and we will only create the object once. Having a constant time to do this can definitely give us the best time performance, but if there is a large number of chances, this can require a large amount of memory. But the second approach can help on that when our number of chances is too high to save some memory and still have a decent time performance.

DEV Community: Miguel Novelo

The Most Interesting Bug I’ve Ever Encountered

The Most Interesting Bug I’ve Ever Encountered

The Symptom: Disk space kept vanishing

My approach: not analyzing threads… but figuring out what the process was actually doing

The clue: a suspicious loop in the call stack

The Root Cause: System.out had been secretly redirected to a file

The effect: infinite output amplification

The hardest part: it wasn’t obvious because it wasn’t in our code

The Fix: restoring System.out as a workaround

Why this bug stands out

Lessons learned

1) Libraries should never modify global output streams

2) Feedback loops can appear from “reasonable” code

3) Thread dumps are not just for concurrency

Final thought

Improving Code Clarity with Functional Programming in Java.

So, What happened?

Understanding the Concepts.

Root Cause

The Most Important Question: Why this wasn't caught in Unit Tests ?

Possible solutions:

Using Lambdas

Creating a new method to be applied in map

Conclusion

Setting probability to choose something randomly

Approach 1 Creating a list with all the chances

Approach 2 Creating a list with the cumulative sum of chances.

Rethinking our solution:

Conclusion

The Root Cause: `System.out` had been secretly redirected to a file

The Fix: restoring `System.out` as a workaround

Creating a new method to be applied in `map`