The Rails Bug That Disappeared After a Server Restart — Until I Found the Mutation

This is a submission for DEV's Summer Bug Smash: Smash Stories.

Project Overview

Some bugs are difficult because the code is complicated.

Others are difficult because the bug seems to have no rules.

This was the second kind.

A Rails application I was working on started rendering ActiveAdmin menus incorrectly.

Sometimes nested menu items appeared under the wrong level.

Sometimes everything looked completely normal.

And sometimes restarting the application made the problem disappear.

That last part was the most suspicious.

A restart shouldn't "fix" a deterministic piece of Ruby code.

So I started digging.

What I eventually found was a much more interesting problem:

An object was being modified somewhere I didn't expect it to be modified.

And the fix turned out to be one of the simplest Ruby changes I've made:

options = options.dup

But getting there took considerably more work.

Bug Fix or Performance Improvement

The application was using ActiveAdmin for its administrative interface.

The problem appeared in the navigation menu.

We had nested menu items, and occasionally they were rendered as top-level items instead of being nested under their intended parent.

Something like:

Expected:

Reports
  ├── Sales
  └── Revenue

Actual:

Reports
Sales
Revenue

The frustrating part was that the behavior wasn't consistently reproducible.

The same application could:

• render the menu correctly
• render it incorrectly
• restart
• start working again
• continue working for some time
• then fail again

This was not the kind of bug where I could simply open the relevant controller and find a typo.

It looked almost random.

The First Clue: Restarting the Server "Fixed" It

The most interesting observation was that restarting the server temporarily resolved the problem.

That immediately made me suspicious of state.

If restarting the process changes the behavior without changing the database or application code, something may be surviving longer than expected inside the process.

That led me away from asking:

"What condition makes the menu render incorrectly?"

and toward:

"What state is being mutated between requests?"

That was a much better question.

Following the Menu Construction

I started tracing how ActiveAdmin constructs its menus.

Eventually I reached the menu node implementation and the add method responsible for adding menu items.

The important part looked roughly like this:

def add(options)
  # menu item construction
end

At first, there was nothing obviously wrong.

The method received an options object and used it to construct the menu node.

But the more I followed the execution path, the more suspicious the object itself became.

The same options object was being passed through multiple parts of the menu-building process.

And something was changing it.

The Problem Wasn't the Value

This was the key realization.

The problem wasn't necessarily that the options contained the wrong value.

The problem was that the options object itself was mutable.

Imagine this simplified example:

options = {
  parent: "Reports",
  label: "Revenue"
}

You pass that object into another method:

build_menu(options)

If build_menu modifies it:

options.delete(:parent)

the original object has now changed too.

There is no copy.

There is no isolation.

Both pieces of code are holding a reference to the same object.

So later:

options[:parent]

returns:

nil

even though the code that originally created the options never intentionally removed it.

That is exactly the kind of mutation that can create a bug that looks random.

Why It Looked Random

The menu wasn't necessarily broken every time.

The behavior depended on when and how the shared object was mutated.

That made the symptoms particularly confusing.

A simplified lifecycle looked like this:

Create menu options
        ↓
Pass options to ActiveAdmin
        ↓
ActiveAdmin modifies options
        ↓
Original object is now different
        ↓
Another menu operation uses it
        ↓
Parent information is missing/incorrect
        ↓
Nested item becomes top-level

And because the object lived inside the running Ruby process, restarting the server cleared that in-memory state.

That explained why a restart could appear to "fix" the bug.

It wasn't fixing anything.

It was simply giving us a fresh process with fresh objects.

Then I Found the GitHub Issue

After spending a significant amount of time debugging the behavior at the application and dependency level, I searched GitHub for similar ActiveAdmin problems.

That's when I found:

ActiveAdmin issue #8078 — "Nested menu items are rendered top-level."

The issue description was remarkably similar.

It reported that nested menus were sometimes rendered at the top level, and that restarting the server temporarily solved the problem. The issue was difficult to reproduce and had been observed in production.

That was the moment when the investigation changed direction.

I wasn't dealing with some mysterious Rails rendering bug.

There was already evidence that ActiveAdmin's menu construction could be modifying menu state.

The issue eventually led to ActiveAdmin PR #8132, titled:

"Make sure menu creation does not modify menu options."

The ActiveAdmin maintainer later reported that the patch had run in production for almost a month without the menu rendering problem recurring before merging it.

Now I had a much stronger hypothesis:

The options object was being mutated during menu construction.

The next question was:

How can I prevent that mutation from affecting the object my application is using?

Code

The Fix: Give add Its Own Copy

Instead of passing the original options object into the existing implementation, I created a duplicate first.

The patch was:

module MenuNode
  def add(options)
    options = options.dup
    super(options)
  end
end

ActiveAdmin::Menu.class_eval do
  include MenuNode
end

That's it.

The important line is:

options = options.dup

Before the fix:

Application
    │
    ▼
options ───────────────┐
                      │
                      ▼
                ActiveAdmin
                      │
                      ▼
                modifies object
                      │
                      ▼
            original options changed

After the fix:

Application
    │
    ▼
original options

    │
    │ dup
    ▼

copied options
    │
    ▼
ActiveAdmin
    │
    ▼
can modify its copy

The original object remains untouched.

Why dup Fixed It

Ruby objects are references.

Consider:

options = {
  parent: "Reports",
  label: "Revenue"
}

copy = options

copy isn't a new Hash.

Both variables point to the same object.

So:

copy.delete(:parent)

also changes:

options

because they're the same object.

But:

copy = options.dup

creates a separate Hash.

Now:

copy.delete(:parent)

doesn't remove the key from the original options.

That's exactly the isolation we needed.

Why I Used super

There was another important detail in the fix.

I didn't want to reimplement ActiveAdmin's add method.

That would have created another maintenance problem.

Instead, I wanted to change one thing:

Give the original method a safe copy of the options.

Then let ActiveAdmin continue doing everything else exactly as it already did.

That's why the implementation is:

def add(options)
  options = options.dup
  super(options)
end

The flow becomes:

Our wrapper
    ↓
duplicate options
    ↓
ActiveAdmin's original add
    ↓
existing behavior

This is a useful pattern when working with a third-party library:

Change the boundary, not the library's internal behavior.

My Improvements

The Debugging Lesson: Don't Trust "Random"

One of the biggest lessons from this bug was that "random" doesn't necessarily mean random.

When an application behaves differently after a restart, there is often state involved.

That state could live in:

• class variables
• global objects
• memoized values
• caches
• singleton instances
• mutable configuration
• shared Hashes or Arrays
• library objects that survive across requests

In Ruby, mutable objects make this especially important.

This:

options = original_options

and this:

options = original_options.dup

look similar.

But they create very different ownership semantics.

The Dependency Was Part of the Application

Another lesson was about debugging third-party libraries.

It is tempting to say:

"The bug is in ActiveAdmin."

But that doesn't help when you're responsible for the application.

The useful question is:

Where does the incorrect state enter my application?

That led me through:

Application behavior
       ↓
ActiveAdmin
       ↓
Menu construction
       ↓
MenuNode#add
       ↓
options object
       ↓
unexpected mutation

Once I could see the mutation boundary, the fix became straightforward.

The Fix Was Tiny. The Investigation Wasn't.

This is something I've noticed repeatedly when debugging production Rails applications.

The final patch often looks deceptively simple.

In this case:

options = options.dup

is almost trivial.

But the path to that line involved:

• reproducing an intermittent problem
• comparing behavior before and after a restart
• tracing ActiveAdmin internals
• inspecting how menu nodes were constructed
• questioning object ownership
• searching existing GitHub issues
• finding a closely related ActiveAdmin issue
• understanding why the existing implementation could mutate shared state
• introducing a minimal wrapper
• delegating back to the original implementation with super

The difficulty wasn't writing the fix.

The difficulty was finding the right layer to fix.

What This Taught Me About Ruby

Ruby makes it very easy to pass objects around.

That's one of its strengths.

But that convenience also means that mutable objects can cross boundaries without making ownership obvious.

Whenever I see a method receiving a Hash or Array, I now ask:

Who owns this object, and is this method allowed to modify it?

If the answer isn't clear, copying the object at the boundary can sometimes be the difference between stable behavior and an intermittent production bug.

Of course, dup isn't a universal solution.

It is a shallow copy, so nested mutable objects can still be shared:

options = {
  menu: {
    parent: "Reports"
  }
}

In that situation:

copy = options.dup

duplicates the outer Hash, but the nested :menu Hash is still shared.

So the correct solution depends on what the called code actually mutates.

In this case, duplicating the options object at the add boundary was enough.

The Bigger Lesson

When a bug looks random, don't immediately assume the framework is behaving unpredictably.

Look for state.

When restarting the server makes the bug disappear, don't celebrate.

Ask what state the restart destroyed.

When a dependency receives a mutable object, don't assume it will leave it untouched.

And when you finally identify the problem, resist the temptation to rewrite everything.

Sometimes the safest fix is simply:

options = options.dup
super(options)

A tiny boundary.

A separate object.

No accidental mutation.

And suddenly the "random" bug isn't random anymore.

---

Final Takeaway

The most valuable part of debugging isn't always the final code change.

Sometimes it's the moment when a confusing symptom finally becomes a predictable consequence of one small assumption.

In my case:

"Menus randomly render incorrectly"
              ↓
"Restarting fixes it"
              ↓
"There must be state"
              ↓
"ActiveAdmin is modifying menu options"
              ↓
"The same mutable object is being reused"
              ↓
"Give ActiveAdmin a copy"
              ↓
options.dup

The final fix was one line.

The investigation was the real engineering work.

And that's probably the part of this bug I'll remember.