DEV Community

Cover image for Debugging Java and Tomcat in Production: A Systematic Approach
matin kaboli
matin kaboli

Posted on

Debugging Java and Tomcat in Production: A Systematic Approach

#debugging #devops #java #performance

When a Java application running on Apache Tomcat becomes slow, starts returning errors, or stops responding, restarting the service is often the easiest option.

But a restart doesn't explain what happened.

The real challenge is identifying whether the problem comes from the JVM, the application, the operating system, the network, or an external dependency.

In this article, we'll walk through a practical workflow for investigating Java and Tomcat issues on a Linux server.

1. Start with the Symptoms

Before changing JVM options or restarting Tomcat, establish what is actually failing.

Is the application completely unavailable, or are requests simply taking longer than usual? Are all endpoints affected, or only a specific operation? Did the problem begin after a deployment, configuration change, or traffic increase?

Start by checking the service:

sudo systemctl status tomcat
Enter fullscreen mode Exit fullscreen mode

The service name may differ depending on the installation.

Find the Java process:

pgrep -af 'org.apache.catalina.startup.Bootstrap'
Enter fullscreen mode Exit fullscreen mode

Check the latest service logs:

sudo journalctl -u tomcat --since "30 minutes ago"
Enter fullscreen mode Exit fullscreen mode

If Tomcat writes to files instead of the system journal, inspect the configured log directory, including catalina.out where applicable.

Look for exceptions, repeated errors, failed database connections, thread pool warnings, and messages around the time the problem started.

The goal is to establish a timeline before changing anything.

2. Check CPU, Memory, and Disk Pressure

A Java process can be running while the application is effectively unavailable.

Start with the host:

uptime
free -h
df -h
df -i
Enter fullscreen mode Exit fullscreen mode

These commands help identify system load, memory pressure, full filesystems, and inode exhaustion.

For process-level CPU and memory usage:

top -p "$(pgrep -fo 'org.apache.catalina.startup.Bootstrap')"
Enter fullscreen mode Exit fullscreen mode

If multiple matching Java processes exist, verify the PID manually before using process-specific diagnostic commands.

For disk I/O, if sysstat is installed:

iostat -xz 1 5
Enter fullscreen mode Exit fullscreen mode

High CPU usage does not automatically mean the JVM needs more CPU. Likewise, low CPU usage does not rule out a serious application problem: threads may be blocked waiting for a database, a lock, or network I/O.

Correlate these observations with application metrics and logs.

3. Inspect the JVM Before Changing Heap Settings

Memory problems are often more complicated than simply increasing -Xmx.

Check the JVM version:

java -version
Enter fullscreen mode Exit fullscreen mode

Identify the Java executable used by the running process, especially when multiple JDK installations exist:

PID=<tomcat_java_pid>

readlink -f "/proc/$PID/exe"
Enter fullscreen mode Exit fullscreen mode

Inspect the JVM command line:

tr '\0' ' ' < "/proc/$PID/cmdline"
echo
Enter fullscreen mode Exit fullscreen mode

If the matching JDK tools are available and you have the necessary permissions, inspect the running JVM:

jcmd "$PID" VM.version
jcmd "$PID" VM.flags
jcmd "$PID" GC.heap_info
Enter fullscreen mode Exit fullscreen mode

Tool availability and supported commands depend on the JDK version and installation.

Heap usage is not the same as process memory

Keep these measurements separate:

  • Java heap: Memory managed by the garbage collector.
  • Metaspace: Memory used for class metadata on modern HotSpot JVMs.
  • Native memory: Memory used by the JVM, threads, direct buffers, native libraries, and other components.
  • RSS: Physical memory currently resident for the process.

A process can consume substantial RAM even when its Java heap is not full.

Before increasing -Xmx, check the container or host memory limit, actual heap usage, garbage collection behavior, and native memory consumption.

For a production system, don't generate a heap dump or a large diagnostic artifact without considering disk space, latency, and the sensitivity of the data it may contain.

4. Investigate Slow or Blocked Requests with Thread Dumps

When Tomcat is running but requests are slow or stuck, thread dumps can help reveal what application threads are doing.

Capture a thread dump using the matching JDK:

jcmd "$PID" Thread.print > /tmp/tomcat-threads.txt
Enter fullscreen mode Exit fullscreen mode

Repeat the capture after a short interval if needed:

sleep 10

jcmd "$PID" Thread.print > /tmp/tomcat-threads-2.txt
Enter fullscreen mode Exit fullscreen mode

Compare the dumps rather than relying on a single snapshot.

Look for patterns such as:

  • Many request threads waiting on database connections.
  • Repeated stack traces pointing to the same slow operation.
  • Threads blocked on Java monitors or locks.
  • Long-running I/O operations.
  • A large number of active threads with little progress.

A thread in WAITING or BLOCKED state is not automatically a problem. Interpret the state together with its stack trace, surrounding threads, and application behavior.

Thread dumps can contain internal class names, SQL statements, URLs, or other sensitive information. Sanitize them before sharing.

5. Check Tomcat Connector and Thread Pool Saturation

Tomcat's request-processing capacity depends on connector configuration and the work performed by each request.

For a configured HTTP connector, inspect relevant settings such as:

  • maxThreads
  • minSpareThreads
  • acceptCount
  • Connection and request timeouts

The exact available settings depend on the connector and Tomcat version.

Increasing maxThreads is not automatically a fix. If requests are blocked waiting for a database connection, creating more request threads may increase contention and memory usage.

Check whether the application is exhausting a JDBC connection pool or waiting on another constrained dependency.

Useful evidence includes active and idle connection counts, request latency, queue depth, timeout rates, and thread pool utilization.

6. Treat Java Agents and Configuration Changes as Variables

Monitoring agents, Java agents, instrumentation libraries, and JVM options can affect application startup and runtime behavior.

If a problem begins after adding an agent or changing the Java runtime, establish exactly what changed.

Record:

  • The previous and current JDK versions.
  • The agent version and startup arguments.
  • The relevant configuration changes.
  • The startup logs and any JVM fatal error reports.
  • Whether the problem can be reproduced in a controlled environment.

For example, an application that fails immediately after a Java upgrade requires a different investigation from one that starts successfully but gradually becomes unresponsive.

Avoid changing several variables simultaneously. Otherwise, it becomes difficult to identify which change caused or resolved the issue.

If the JVM produces a fatal error report, inspect the report and associated crash logs. A native JVM crash is different from an ordinary Java exception and should not be diagnosed from application logs alone.

7. Verify the Hypothesis Before Calling It a Fix

Once you have a plausible root cause, make the smallest safe change that tests it.

Before applying a production change, consider its potential impact, rollback procedure, and whether a restart is required.

Afterward, compare the same indicators you collected before the change:

  • Request latency and error rate.
  • CPU and memory usage.
  • Garbage collection behavior.
  • Thread pool utilization.
  • Database connection pool metrics.
  • Relevant application logs.

A successful restart is not proof that the root cause has been fixed. The issue may return under the same workload.

A Practical Incident Checklist

When a Tomcat application becomes slow or unavailable, I would organize the investigation around these questions:

  1. What changed immediately before the incident?
  2. Is the issue at the host, JVM, Tomcat, application, or dependency layer?
  3. Are CPU, memory, disk, or network resources constrained?
  4. Are request threads progressing, blocked, or waiting?
  5. Is the JVM healthy, and is garbage collection behaving as expected?
  6. What evidence supports the suspected root cause?
  7. How will the fix be validated, and how can it be rolled back?

Final Thoughts

Production debugging is not about collecting as many logs as possible or applying JVM tuning options until the symptoms disappear.

It is about gathering evidence, narrowing the problem to the right layer, testing a hypothesis, and verifying the result.

The most useful diagnostic command is the one that answers a specific question about the system—not necessarily the one with the most output.

I'm interested in Linux infrastructure, JVM behavior, distributed systems, and the operational practices that make services more reliable. I'll continue sharing practical technical notes and investigation workflows as I work on these topics.

What do you usually investigate first when a Tomcat application becomes unresponsive: JVM metrics, thread dumps, application logs, or external dependencies?

Top comments (0)