DEV Community

Cover image for 🧵 From JRuby + WebLogic Panic to Peace: A Debugging Story
Santanu Bhattacharya
Santanu Bhattacharya

Posted on

🧵 From JRuby + WebLogic Panic to Peace: A Debugging Story

#devjournal #java #ruby #weblogic

How a tiny git: in Gemfile, cache broke our WAR deployment — and cost me a sleepless night.

😰 The Situation

We have multiple JRuby applications deployed on Oracle WebLogic.
One of them suddenly stopped working.

It had been:

  • ✅ Working in production
  • ✅ Working earlier on this same server
  • ❌ Suddenly failing during startup The error: 
org.jruby.exceptions.StandardError:
(GitError) https://github.com/jruby/warbler (at master@973d14c)
is not yet checked out. Run `bundle install` first.
Enter fullscreen mode Exit fullscreen mode

And not just one error, And not just one error.
Multiple layers of errors.
That’s when the tension started.

🥊 Phase 1: Class Conflict Error

The first error looked like a classic classloader issue:

undefined method `bytesize' for Rack::Utils:Module
Enter fullscreen mode Exit fullscreen mode

That’s usually a symptom of:
Different Rack versions loaded
Library collision
Parent-first classloading override

We had:
Multiple JRuby apps in the same WebLogic domain
Different JRuby versions (The other app is using 9.3.5.0 and this one 9.4.5.0)
Different Rack versions
WebLogic loads classes parent-first by default.
That means server-level classes may override WAR classes.
So if another JRuby app loads Rack first…
You’re in trouble.

One solution is, we can downgrade the application to 9.3.5.0 and try redeploying to check if it works. But at first, I don't want to downgrade the application as earlier I have spent a lot to upgrade this application.

Fix Attempt #1 — weblogic.xml Change

Since we have other Jruby applications in other managed server. When we compare the weblogic.xml, it has different content than what I maintained earlier. So I bring that part of the xml here as well.

We updated weblogic.xml:

<weblogic-web-app>
  <container-descriptor>
    <prefer-web-inf-classes>true</prefer-web-inf-classes>
  </container-descriptor>
</weblogic-web-app>
Enter fullscreen mode Exit fullscreen mode

This tells WebLogic:

  • Load classes from the WAR first.
  • Since the other application's JRuby version is different, it somehow still gives the same error after deployment.

Also, I found that the jar file of the application is still different in the WebLogic log. And the application is failing at the weblogic level. The application log has not been hit yet.

I cross check the Gemfile.lock and found it is different than 9.4.5.0. So, I decided to update that to same 9.4.5.0.
But how the application was working early in weblogic?

Too Many Deployments

If you have multiple versions of WAR deployed (including old ones not yet undeployed), WebLogic validates all of them before displaying the UI. So I decided to clean the cache if there is any in WebLogic.

Fix:
I have found the cache and tmp directories.
Clean staging directory:

  • $DOMAIN_HOME/servers/AdminServer/tmp
  • $DOMAIN_HOME/servers/AdminServer/cache

(Stop server first)
After deploying with this change:
✅ The class conflict error disappeared.
Relief?
Not yet.
Because something worse appeared.

💥 Phase 2: The Git/Bundler Explosion

Now the error became:

(GitError) https://github.com/jruby/warbler (at master@973d14c)
is not yet checked out. Run `bundle install` first.
Enter fullscreen mode Exit fullscreen mode

Which makes zero sense in production.

You don’t run bundle install inside a deployed WAR.
WAR files should be fully packaged artifacts.
So why was Bundler complaining?

The Real Clue: Expanding the WAR

I extracted the WAR:
jar xf myapp.war

Inside:

/WEB-INF/gems/bundler/gems/

I saw:

warbler-973d14c8f7e3

That suffix (973d14c8f7e3) means:

Warbler was installed from Git.
And then I checked our Gemfile.

The Actual Root Cause

We had this in Gemfile:

gem 'warbler', '2.0.5', git: 'https://github.com/jruby/warbler', branch: 'master', platforms: :jruby

This means:

Warbler was pinned to a Git commit
Bundler stored commit metadata
The WAR expected a git checkout

But in WebLogic:

No .git
No working tree
No git context
So Bundler threw:
is not yet checked out
It worked before because:

  • Cached bundle
  • Slightly different build state

But it was fragile.
And eventually, it broke.
I checked in web and found that they are suggesting to use the gem from the rubygems.org instead of git. My other applications are working fine. So I decided to keep it simple and pull from github only.

✅ The Correct Fix

The WAR contained:
/WEB-INF/gems/bundler/gems/warbler-973d14c8f7e3

That commit hash indicated:
Bundler had cached a git checkout

The artifact embedded metadata tied to a specific commit
Something in that cached state became inconsistent

Because I wasn’t rebuilding with a clean bundle, the WAR kept packaging a partially cached or inconsistent git state.
The real solution was:

Use the github commit:

gem 'warbler', git: 'https://github.com/jruby/warbler.git', ref: '93d14c'
Enter fullscreen mode Exit fullscreen mode

Then rebuild clean:

bundle install --no-cache

That forced Bundler to:

  • Re-fetch the git repository
  • Re-resolve dependencies
  • Remove stale metadata Rebuild cleanly

RAILS_ENV=production bundle exec jruby -S warble --trace
Also this time, I restarted the weblogic service from the server entirely.
Redeploy.

Everything worked.

What Actually Happened

  1. Classloader conflict caused Rack error
  2. Fixed via prefer-web-inf-classes
  3. That exposed a git-based Warbler inconsistency
  4. Cached bundle state caused broken WAR packaging
  5. bundle install --no-cache cleaned it
  6. Rebuild fixed everything
  7. This wasn’t a single bug.

It was layered:
Server classloading
Git-based gem
Bundler caching behavior

Top comments (0)