A payment callback fails intermittently. Ops searches Kibana for
orderId=202611180001and comes up empty, so they have to SSH in and grep through 400,000 lines of logs. The reason isn't complicated: the log is a single line of plain text, and the middleware only sees one long string of characters — there's noorderIdfield. This post lays out Spring Boot's built-in JSON logging clearly: three config changes, no third-party dependencies, stack traces still intact, and once you're done you can click through ELK and filter by field.
1. Why plain-text logs can't be found in ELK
The root problem with plain-text logs is that there are no fields — just one line of characters. When the machine gets that line, it has no idea which part is the order number, which is the user ID, and which is the elapsed time.
A typical line looks like this:
2026-11-18 03:12:41.882 ERROR [http-nio-8080-exec-7] c.e.logdemo.OrderService - pay callback failed, orderId=A202611180001, userId=88213, cost=3120ms
Anyone can read that at a glance. But drop it into Elasticsearch and the default standard analyzer splits on punctuation: orderId, A202611180001, cost, 3120ms all get chopped into a pile of independent tokens. Want to find "logs where the order number equals A202611180001"? You have to hack it together with match_phrase and quotes — and the moment there's a Chinese character or an equals sign in there, it stops matching. Aggregation is even worse — count failures by userId? There's no such field, so aggregation isn't even on the table.
Let me break down three common misconceptions one by one.
Misconception 1: assuming MDC automatically becomes fields.
MDC just stuffs key-value pairs into the current thread's ThreadLocal; at output time, %X{traceId} in the pattern pulls them out and splices them into the string. After splicing, it's still one line of text. Only in JSON logs does MDC become real fields — these two things get conflated all the time.
Misconception 2: writing grok regexes in Logstash to brute-force it.
Regex can parse it, but it's brittle. Someone adds one extra comma in the business code, or a field value contains a space, and the regex matches off-target — the whole log line becomes _grokparsefailure. On top of that, parsing costs real-time CPU, and once QPS climbs, the Logstash box is the first to fall over.
Misconception 3: building JSON yourself with StringBuilder.
Log messages contain double quotes, newlines, exception stack traces, and Chinese characters. Hand-assembling JSON means dealing with escaping and with the newlines inside stack traces. Hand-rolled assembly code produces invalid JSON the first time a message contains a quote, the whole log line gets dropped by the collector, and you won't even know it was dropped.
Here's how the two kinds of logs differ along the collection pipeline:
| Aspect | Plain text + grok | Native JSON |
|---|---|---|
| Field recognition | Guessed by regex; breaks the moment the pattern changes | The collector parses it directly with json
|
| Precise filtering | Only fuzzy matching on the whole line | orderId: "A202611180001" |
| Aggregation | Basically impossible | group by any field |
| Stack traces | Multi-line text; the collector needs multiline config | A single field, valid by construction |
| Field-name consistency | Depends on people being disciplined | The framework outputs them uniformly |
2. How Spring Boot structured logging actually outputs
Spring Boot's structured logging ships with three built-in JSON formats: ECS, GELF, and Logstash. No third-party dependencies — just change the config and it works.
Let's start with the full path a log line takes from code to file. Once you understand that path, you can derive every config option yourself.
Step 1: business code calls log.info(...). SLF4J wraps that call in a ILoggingEvent carrying the timestamp, log level, logger name, thread name, formatted message, MDC snapshot, and exception object.
Step 2: the Appender receives the event and hands it to the Encoder. The Encoder's job is to turn the event into a byte array and write it out. In a traditional setup that's PatternLayoutEncoder, which does string substitution against a pattern string — that's where text logs come from.
Step 3: Spring Boot swaps out the Encoder. You set logging.structured.format.console=ecs, Spring Boot sets the CONSOLE_LOG_STRUCTURED_FORMAT system property to ecs, and using its bundled logback config, it replaces the console appender's encoder with org.springframework.boot.logging.logback.StructuredLogEncoder.
Step 4: The Encoder looks up the Formatter by format. StructuredLogEncoder takes the ecs id, finds the corresponding formatter, and serializes the same ILoggingEvent into a single line of JSON. All key-value pairs in the MDC get flattened into the top level of the JSON, and the exception object gets serialized into a field like error.stack_trace.
Step 5: Write one entry per line. This is the most critical constraint for JSON logging: a log entry must end within a single line, with no bare line breaks. Line breaks in stack traces get escaped as \n. Collectors (Filebeat, Fluent Bit) read files line by line by default, so one JSON entry per line lets them parse directly with json; if a stack trace is spread across dozens of lines, the collector has to be configured with multiline rules to stitch it back together — and one misconfiguration and a log entry falls apart.
The field differences across the three formats come down to this table, which is exactly what you should look at first when choosing:
| Item | ECS | Logstash | GELF |
|---|---|---|---|
| Time field |
@timestamp, ISO-8601 nanoseconds |
@timestamp, with timezone offset |
timestamp, floating-point seconds |
| Message field | message |
message |
short_message |
| Level |
log.level string |
level + level_value numeric |
level numeric (6 = INFO) |
| Service name |
service.name, defaults to spring.application.name
|
none |
host, defaults to spring.application.name
|
| Primary consumer | Elasticsearch / Kibana | Logstash / Filebeat | Graylog |
ECS is Elastic's own field spec, where field paths are nested objects, e.g. log.level, process.thread.name. GELF is Graylog's format, and level uses the syslog-style numbers. The Logstash format is the plainest — flat fields, no nesting.
3. Hands-On: Three Steps to JSON Logs
Step 1: Dependencies and Configuration
pom.xml, complete and ready to use:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.0.9</version>
<relativePath/>
</parent>
<groupId>com.example</groupId>
<artifactId>log-demo</artifactId>
<version>1.0.0</version>
<name>log-demo</name>
<properties>
<java.version>21</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webmvc</artifactId>
<version>4.0.9</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>4.0.9</version>
</plugin>
</plugins>
</build>
</project>
Here we use spring-boot-starter-webmvc rather than spring-boot-starter-web. The latter is already marked deprecated in its POM description, and the official guidance points to the new webmvc starter. Both work, but for new projects just use webmvc.
Main application class:
package com.example.logdemo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class LogDemoApplication {
public static void main(String[] args) {
SpringApplication.run(LogDemoApplication.class, args);
}
}
application.yml:
spring:
application:
name: log-demo
version: 1.0.0
server:
port: 8080
logging:
file:
name: logs/app.json
structured:
format:
console: ecs
file: logstash
ecs:
service:
environment: prod
node-name: node-1
json:
add:
team: order-platform
stacktrace:
max-length: 4096
Let's go through each item.
logging.structured.format.console controls console output and format.file controls file output. The two can differ: the console is read by the collection agent on the ELK side, so use ECS; the file is written to disk for archiving, so use the Logstash format. This isn't showing off — plenty of teams collect from stdout on the console and keep files for audit.
If logging.structured.ecs.service.name is omitted, it defaults to spring.application.name. But the service name alone isn't enough: when multiple environments share one Kibana, environment and node-name are the key to telling prod from staging, so set them too.
logging.structured.json.add adds fixed fields. team: order-platform will appear in every log entry, and on the ES side it can be used for per-team data isolation or index routing.
logging.structured.json.stacktrace.max-length limits the maximum length of a serialized stack trace. By default, the entire stack trace is stuffed into JSON; with deep nesting, a single log entry can reach tens of KB, wasting both disk space and transfer bandwidth.
Step 2: Put request context into MDC
This is where JSON logging really pays off: request-level fields automatically go into every log entry, so you don’t have to pass parameters repeatedly in every log.info line.
package com.example.logdemo.web;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.slf4j.MDC;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import java.io.IOException;
import java.util.UUID;
@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
public class TraceIdFilter extends OncePerRequestFilter {
@Override
protected void doFilterInternal(HttpServletRequest request,
HttpServletResponse response,
FilterChain filterChain) throws ServletException, IOException {
String traceId = request.getHeader("X-Trace-Id");
if (traceId == null || traceId.isBlank()) {
traceId = UUID.randomUUID().toString().replace("-", "");
}
MDC.put("traceId", traceId);
MDC.put("path", request.getRequestURI());
MDC.put("clientIp", resolveClientIp(request));
response.setHeader("X-Trace-Id", traceId);
try {
filterChain.doFilter(request, response);
} finally {
MDC.clear();
}
}
private String resolveClientIp(HttpServletRequest request) {
String forwarded = request.getHeader("X-Forwarded-For");
if (forwarded != null && !forwarded.isBlank()) {
return forwarded.split(",")[0].trim();
}
return request.getRemoteAddr();
}
}
You can’t skip MDC.clear() in finally. Tomcat threads are reused; if you don’t clear it, the traceId from the previous request gets carried into the next one, and all your logs get mixed up. This kind of pitfall is very painful to troubleshoot because the logs appear to “have a traceId,” but it’s attached to the wrong request.
This filter is a @Component; Spring Boot automatically registers it as a Servlet Filter, and @Order(HIGHEST_PRECEDENCE) ensures it runs before your business code, so the first log line entering the Controller already carries traceId.
Service class and Controller:
package com.example.logdemo.order;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
@Service
public class OrderService {
private static final Logger log = LoggerFactory.getLogger(OrderService.class);
public String pay(String orderId, long amount) {
long begin = System.currentTimeMillis();
log.atInfo()
.addKeyValue("orderId", orderId)
.addKeyValue("amount", amount)
.log("pay start");
try {
if (amount <= 0) {
throw new IllegalArgumentException("amount must be positive: " + amount);
}
if (orderId != null && orderId.startsWith("FAIL")) {
throw new IllegalStateException("payment gateway timeout, orderId=" + orderId);
}
long cost = System.currentTimeMillis() - begin;
log.atInfo()
.addKeyValue("orderId", orderId)
.addKeyValue("costMs", cost)
.log("pay success");
return "SUCCESS";
} catch (RuntimeException e) {
log.atError()
.addKeyValue("orderId", orderId)
.setCause(e)
.log("pay failed");
throw e;
}
}
}
log.atInfo().addKeyValue(...) is SLF4J 2’s fluent API. Key-value pairs added to it apply only to that single log entry and automatically appear at the top level of the JSON. The difference from MDC is scope: MDC covers the whole thread and the whole request, while addKeyValue covers only that one entry. For values like an order ID, which has one value per call, addKeyValue is more appropriate.
setCause(e) is much better than concatenating the exception into the message string. The former lets the framework serialize the exception object into a separate error.stack_trace field that can be expanded in Kibana; the latter is just a blob of text with the stack trace and message mashed together, making it impossible to aggregate by exception type.
package com.example.logdemo.order;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/orders")
public class OrderController {
private final OrderService orderService;
public OrderController(OrderService orderService) {
this.orderService = orderService;
}
@GetMapping("/{orderId}/pay")
public String pay(@PathVariable String orderId,
@RequestParam(defaultValue = "100") long amount) {
return orderService.pay(orderId, amount);
}
}
Global exception handling ensures that thrown exceptions also leave structured logs:
package com.example.logdemo.web;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import java.util.Map;
@RestControllerAdvice
public class GlobalExceptionHandler {
private static final Logger log = LoggerFactory.getLogger(GlobalExceptionHandler.class);
@ExceptionHandler(Exception.class)
public ResponseEntity<Map<String, String>> handle(Exception e) {
log.atError()
.addKeyValue("exceptionType", e.getClass().getSimpleName())
.setCause(e)
.log("unhandled exception");
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
.body(Map.of("message", "internal error"));
}
}
After startup, send a request:
curl -i "http://localhost:8080/orders/A202611180001/pay?amount=100"
curl -i "http://localhost:8080/orders/FAIL-001/pay?amount=100"
The console output is one JSON object per line, with fields roughly like this (I’ve arranged them according to the ECS structure so you can compare):
{"@timestamp":"2026-11-18T03:12:41.882Z","log":{"level":"INFO","logger":"com.example.logdemo.order.OrderService"},"process":{"pid":48213,"thread":{"name":"http-nio-8080-exec-7"}},"service":{"name":"log-demo","version":"1.0.0","environment":"prod","node":{"name":"node-1"}},"team":"order-platform","traceId":"9f3c1a5e7b2d4c60a8e1f0b3d5c7a9e2","path":"/orders/A202611180001/pay","clientIp":"127.0.0.1","orderId":"A202611180001","amount":100,"costMs":2,"message":"pay success"}
A quick comparison shows what you get for free: traceId, path, and clientIp come from MDC; orderId, amount, and costMs come from addKeyValue; team comes from configuration; and service.* and process.* are filled in automatically by the framework. There isn’t a single line of string concatenation in the business code.
Step 3: Customize the logback configuration while keeping it readable locally
JSON logs are basically unreadable in docker logs. Someone on the team will always want to read logs directly in the local console, and switching formats by environment is the easiest way to handle that.
logback-spring.xml, the full file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property name="LOG_PATH" value="logs"/>
<springProfile name="dev,local">
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} %-5level [%thread] [%X{traceId}] %logger{36} - %msg%n</pattern>
<charset>UTF-8</charset>
</encoder>
</appender>
</springProfile>
<springProfile name="!dev & !local">
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="org.springframework.boot.logging.logback.StructuredLogEncoder">
<format>${CONSOLE_LOG_STRUCTURED_FORMAT}</format>
<charset>UTF-8</charset>
</encoder>
</appender>
</springProfile>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/app.json</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/app.%d{yyyy-MM-dd}.%i.json.gz</fileNamePattern>
<maxFileSize>100MB</maxFileSize>
<maxHistory>7</maxHistory>
<totalSizeCap>5GB</totalSizeCap>
</rollingPolicy>
<encoder class="org.springframework.boot.logging.logback.StructuredLogEncoder">
<format>${FILE_LOG_STRUCTURED_FORMAT}</format>
<charset>UTF-8</charset>
</encoder>
</appender>
<appender name="ASYNC_FILE" class="ch.qos.logback.classic.AsyncAppender">
<queueSize>2048</queueSize>
<discardingThreshold>0</discardingThreshold>
<neverBlock>false</neverBlock>
<appender-ref ref="FILE"/>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="ASYNC_FILE"/>
</root>
</configuration>
A few notes.
Once you use a custom logback configuration, Spring Boot’s default configuration no longer takes effect, so you need to explicitly switch the encoder to StructuredLogEncoder, with the format taken from the two system properties ${CONSOLE_LOG_STRUCTURED_FORMAT} and ${FILE_LOG_STRUCTURED_FORMAT}—these are exactly the values injected by logging.structured.format.console and logging.structured.format.file. Hardcoding ecs also works, but you lose the flexibility of changing formats in yml.
In the expression for springProfile, & is the XML-escaped form of !dev & !local. The dev and local profiles use colored text, while all other environments use JSON.
The file Appender is wrapped in an AsyncAppender: writing logs to disk is synchronous IO, and a brief disk hiccup can block business threads. A queue size of 2048 is enough for the vast majority of business systems; setting discardingThreshold to 0 means no logs are dropped even when the queue is full (the default is 20, and once the queue reaches 80% usage it starts dropping TRACE, DEBUG, and INFO—this default has tripped up plenty of people).
Four: Six Pitfalls, and I’ve Hit at Least Four
Pitfall 1: MDC is empty in asynchronous threads.
You can’t get MDC in @Async methods, custom thread pools, or CompletableFuture because MDC is stored in ThreadLocal and disappears when the thread changes. Once traceId is empty in the logs, the entire trace is broken. The solution is to configure TaskDecorator for the thread pool: copy the current thread’s MDC into the child thread when the task is submitted, and clean it up after the task finishes. Spring’s ThreadPoolTaskExecutor supports setTaskDecorator, so a one-line change is enough.
Pitfall 2: Phone numbers, national ID numbers, and tokens in logs all get collected.
After switching to JSON, the fields became clear and collection became easier—including collecting things you shouldn't be collecting. Before going live, you must go through the output fields. Sensitive fields should either not be emitted at the source, or filtered out with logging.structured.json.exclude. Sensitive keys in MDC especially should not be included in the filter at all.
Pitfall 3: ECS service.name is the same value across all instances.
spring.application.name does not distinguish instances. After eight nodes write to ES, service.node.name is also empty, so when something goes wrong you can't tell which machine it came from. node-name recommends injecting the Pod name or hostname via an environment variable instead of hardcoding it.
Pitfall 4: Assuming that changing the console also changes the file.
format.console and format.file are two independent switches. If you only configure console, the file is still plain text; the reverse is also true. If you want JSON on both sides, configure both.
Pitfall 5: The collection side treats JSON as text and processes it again.
Filebeat's json parsing is configured in processors. After the message field is processed, remember to drop_fields to delete the original line; otherwise, one log is stored twice in ES and the index doubles in size.
Pitfall 6: Enabling JSON in local development and then starting to question everything.
A single log entry is hundreds of characters, all crammed onto one line, and your eyes simply can't scan it. In the development environment, just use springProfile's text format. Don't force yourself to endure it.
V. How to Choose Among the Three Formats
For selection, look only at who your logging platform is. There is no third criterion.
| Scenario | Recommended Format | Reason |
|---|---|---|
| Elasticsearch + Kibana | ecs |
Field paths fit the Elastic ecosystem, and visualization works out of the box |
| Existing Logstash / Filebeat legacy pipeline | logstash |
Fields are flat, consistent with field names in the legacy pipeline, and no mapping changes are needed |
| Graylog | gelf |
Graylog input only accepts GELF; other formats cannot get in |
If the logging platform has not been decided yet, I recommend ECS. Its field naming is the most standardized, the experience of filtering nested structures in Kibana is clearly better than flat fields, and if you change the collection side later, the field names do not need to be rewritten.
On performance, to be honest: JSON is definitely more expensive than plain text. With more fields, each log entry becomes 20% to 30% larger, and serialization itself also has overhead. But the retrieval capability gap this overhead buys is orders of magnitude—plain-text logs are basically as good as nonexistent when failures occur. If you really want to reduce overhead, two actions are the most effective: truncate the logger name to the package level, and change the file Appender to asynchronous. After compression (.gz), the size gap between the two becomes much smaller, and disk is usually not the bottleneck.
VI. Summary
Changing the log format only takes ten minutes, but it determines whether, when a failure occurs, you "click twice and filter it out" or "grep for forty minutes."
Just remember three configuration points: enable JSON in logging.structured.format.console and format.file, and clearly mark the service and environment in logging.structured.ecs.service.*. Spend the rest of your effort on MDC—put request-level fields into MDC, use addKeyValue for call-level fields, and hand exceptions off with setCause instead of concatenating strings yourself.
The easiest things to miss in the pitfall checklist are MDC.clear() and MDC propagation across asynchronous threads. The former causes log entries to get mixed up, and the latter breaks the trace. Neither throws an error; they only make you question everything while troubleshooting.
Written by Lü Dongbin, who can code
Top comments (0)