Why Log levels matter?

#softwaredevelopment #logging #springboot #java

The Definitive Guide to Log Levels and Centralized Logging

Ever wondered why people dont use system.out to print/log ? Or why at one place someone use log.debug and other places log.info. Why your review comments always ask for more logging statements. Why it matters to choose right logging level. How to effectively use it ?

Effective logging is the backbone of application health monitoring and debugging. By consistently and correctly using log levels, developers can maintain a crucial b*alance between visibility and noise* across all environments.

Understanding Log Levels and Their Usage

Log levels are a hierarchical system used to categorize the severity or importance of a log message. The common hierarchy, from least to most severe, is generally:

TRACE: Very granular information, typically used for detailed tracing of a request or process flow. You’d enable this only for deep, short-term debugging.

1. DEBUG: Fine-grained informational events that are most useful to debug an application. This includes variable values, entry/exit points of methods, and steps taken within a process.

2. INFO: Confirmation that things are working as expected. This level provides general application health and progress messages, such as service startup/shutdown, major state changes, or successful key operations. This is often the default level for production environments.

3. WARN: Potentially harmful situations or unexpected events that might indicate a problem but do not necessarily prevent the application from continuing. Examples include use of deprecated APIs, hitting a soft limit, or a recoverable failure.

4. ERROR: Serious errors that prevent certain parts of the application from functioning, but the application as a whole might still be running. Examples are exceptions that cause a feature to fail.

5. FATAL: Very severe error events that will likely cause the application to abort. This is usually reserved for catastrophic failures that bring down the entire system or a critical component.

WARN vs. INFO: Distinguishing the Difference

Java Logging Anti-Patterns

Why Avoid e.printStackTrace()
Calling e.printStackTrace() directly within application code is highly discouraged:

1. Uncontrolled Output Stream: It writes directly to the** Standard Error Stream (System.err)**, bypassing the configured logging framework (e.g., Log4j, SLF4J).

2. No Filtering or Formatting: The output cannot be formatted (no timestamp, log level) and cannot be filtered.

3. Bypasses Log Destination: It ignores your configuration for centralized systems like Splunk.

The recommended practice is to log the exception using the framework at the ERROR level, ensuring the full stack trace is captured:

// Preferred approach:
try {
// … code that might throw an exception
} catch (Exception e) {
logger.error(“Error processing request X”, e); // The ‘e’ parameter logs the full stack trace
}

The Problem with System.out.println()

Using System.out.println() for application logging is problematic because it

1. Writes to Standard Output Stream (System.out): This bypasses the logging framework entirely, leading to unstructured and untagged output.

2. No Context: The message lacks crucial metadata like the log level, timestamp, or thread name.

3. Inflexible Configuration: You cannot easily control or filter the output without code redeployment.

Sending Logs to Centralized Systems (Splunk, LogicMonitor, etc.)

To leverage the power of Centralized Logging Systems (Splunk, LogicMonitor, ELK Stack), you must configure your logging framework (Log4j, Logback) to direct output to them. This is done via Appenders (or Handlers) in your external configuration file.

1. The Robust Approach: File-Based Logging with an Agent (Recommended)

This is the most common and reliable method, favored by large-scale deployments:

● Application Action: The application uses a standard RollingFileAppender to write highly structured logs (preferably JSON) to a local file.

● Agent Action: A dedicated, lightweight log forwarding agent (e.g., Splunk Universal Forwarder, LogicMonitor Collector, Fluentd) runs on the same server.

● Shipping: The agent is configured to “tail” (monitor) the application’s log file, read new entries in real-time, and forward them over the network to the central log aggregator.

Benefit for Centralized Logging

Reliability Logs are safely stored locally if the network or collector is down, ensuring no data loss.

Performance Writing to a local file is much faster than synchronous network calls.

Parsing The agent can standardize the data into JSON or key-value pairs before ingestion.

2. Direct Network Appenders

For specialized needs, you can configure the logging library to send logs directly over the network:

• Network Protocol Appender: Configure an appender to send logs via a protocol like Syslog (TCP/UDP).

• HTTP/API Appender: Use a specialized appender (like SplunkHttpEventCollectorAppender) to send structured JSON data directly to the vendor’s API endpoint.

Crucially, no modification to the application’s core logging calls is needed. You continue to use standard calls (e.g., logger.info(…)). The logging framework handles the delivery mechanism based on the external configuration file.