Satyaki Saha

Posted on Nov 14 Edited on Nov 14

Why Our Java Memory Crept Up on Flat Traffic: A Modern Stack Post-Mortem

#backend #debugging #java #devops

TL;DR: After upgrading our stack, our Java service's memory usage kept climbing steadily despite flat request volumes. It looked like a single memory leak on our APM dashboard, but digging deeper revealed three separate issues working together: virtual thread payload retention, an unbounded ANTLR 4 static DFA cache in Hibernate 7, and elevated baseline class loading. Here is how we diagnosed and fixed each one.

🔍 The Symptoms

Right after migrating our core service to JDK 25 and Spring Boot 4.1 (which brings in Hibernate 7.4), we noticed our memory graph taking an upward slope:

Memory (MB)
  ^
  |          /---- Spikes (Virtual Threads)
  |         /
  |  /\    /|    /---- Steady Growth (ANTLR 4 DFA Cache)
  | /  \  / |   /
  |/____\/__|__/______ Baseline Shift (38.8k Classes Loaded)
  +-------------------------------------> Time

Request volume was completely flat. Our high-level metrics suggested a classic memory leak, but a single fix didn't resolve it. We were actually fighting three distinct memory behaviors disguised as one issue on our graphs.

🚨 Issue 1: Unbounded Virtual Threads & Memory Spikes

Virtual threads are lightweight, but the payload data they process on the heap is not.

Once we enabled virtual threads without capping concurrency on high-throughput or heavy outbound operations, sudden traffic micro-bursts executed fully in parallel. Every in-flight request retained its object graph on the heap simultaneously.

The Double Hit

Downstream Overwhelm: Downstream services got slammed without backpressure.
Heap Pressure: Heap allocation spiked rapidly, causing aggressive GC pauses during traffic bursts.

While this wasn't responsible for the long-term steady memory growth, it magnified transient spikes.

The Fix: Target Concurrency at the Outbound Layer

Rather than throttling threads globally, cap concurrency where work gets expensive—such as downstream client calls. A bounded-wait Semaphore fails fast so queued requests don't hang around filling up the heap.

package com.example.client;

import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;

@Component
public class DownstreamClient {

    // Size this to downstream capacity
    private final Semaphore permits = new Semaphore(50);
    private final RestClient restClient;

    public DownstreamClient(RestClient.Builder builder) {
        this.restClient = builder.baseUrl("https://downstream.example").build();
    }

    public String fetch(String id) throws InterruptedException {
        if (!permits.tryAcquire(2, TimeUnit.SECONDS)) {
            throw new IllegalStateException("Downstream is saturated, try again later");
        }
        try {
            return restClient.get()
                    .uri("/items/{id}", id)
                    .retrieve()
                    .body(String.class);
        } finally {
            permits.release();
        }
    }
}

💡 Additional Concurrency Controls across Layers

Inbound Traffic: Tune server.tomcat.max-connections and server.tomcat.accept-count.
Queue Consumers: Bound max concurrent messages on SQS/Kafka listeners.
Database Connection Pools: Limit Hikari pool size (remember: waiting threads still hold request objects in memory!).
Spring Framework 7: Check out @ConcurrencyLimit for declarative control.

🕳️ Issue 2: Hibernate 7's ANTLR 4 Static DFA Cache Growth

This was the actual culprit behind the steady long-term memory climb.

How Hibernate Query Caching Works

Hibernate parses HQL/JPQL strings into internal AST trees and caches them in the QueryInterpretationCache. The default capacity is 2048 entries (hibernate.query.plan_cache_max_size).

Several endpoints in our service executed SQL IN queries with dynamically built, variable-length parameter lists:

@Query("select o from Order o where o.id in :ids")
List<Order> findByIds(@Param("ids") Collection<Long> ids);

Passing a collection of 3 elements produced a different query shape than a collection of 5 or 12 elements. This flooded the cache with thousands of unique query structures.

The Hidden ANTLR 4 Gotcha

Under the hood, Hibernate 6 and 7 use ANTLR 4 for parsing HQL (replacing ANTLR 2 used in Hibernate 5).

When we examined our heap dumps in Eclipse MAT, the memory wasn't tied up in Hibernate's query plan cache—it was dominated by:

org.antlr.v4.runtime.atn.DFAState
org.antlr.v4.runtime.atn.*

Key Takeaway: ANTLR 4 maintains a static DFA (Deterministic Finite Automaton) cache shared across the entire JVM. It has no maximum size setting, never evicts entries, and grows indefinitely as the parser encounters new query shapes. hibernate.query.plan_cache_max_size will not cap it!

The Fix

To keep query shapes static regardless of array or collection sizes:

Enable IN-clause padding in your application properties:

   hibernate.query.in_clause_parameter_padding=true

Always bind collections as parameters rather than building dynamic HQL strings.

🏗️ Issue 3: High Baseline Class Loading

Following our migration, our service loaded roughly 38.8k classes at startup. Pulling in dependencies like AWS SQS (which brought in Netty), coupled with framework updates across JDK 25 and Spring Boot 4.1, pushed our baseline off-heap memory usage noticeably higher.

Mitigation vs. Root Cause Fix

Mitigation: Setting strict limits on Metaspace (-XX:MaxMetaspaceSize) and sizing Max RAM percentage (-XX:MaxRAMPercentage) prevents container OOM kills, but it only contains symptoms.
Root Cause Fix: To actually lower off-heap baseline footprint, run dependency analysis tools to prune unused transitives and explicitly disable unneeded Spring auto-configurations (@SpringBootApplication(exclude = ...)).

📊 Summary Diagnostic Playbook

Different memory symptoms demand different diagnostic tools:

Symptom	Primary Suspect	Diagnostic Tool / Strategy	Actionable Fix
Sharp Spikes	Unbounded Concurrency / Payload Retention	Thread Dumps, APM Concurrency Metrics	Apply `Semaphore`, rate-limit consumers
Steady Upward Slope	Dynamic Parser Caches (e.g. ANTLR DFA)	Eclipse MAT / Heap Dump OQL	Enable `in_clause_parameter_padding`
High Elevated Baseline	Excess Class Load & Off-Heap Metaspace	Native Memory Tracking (NMT), `-verbose:class`	Exclude unused dependencies & auto-configs

💬 Discussion

Have you encountered unexpected memory behavior after upgrading to Hibernate 7 or enabling Virtual Threads in production? Let me know down in the comments!

DEV Community