DEV Community: Bishwa Poudel

Advanced Error Handling in Spring Boot Microservices

Bishwa Poudel — Mon, 28 Oct 2024 06:54:06 +0000

In complex microservices, advanced error handling goes beyond simple exception logging. Effective error handling is crucial for reliability, scalability, and maintaining good user experience. This article will cover advanced techniques for error handling in Spring Boot microservices, focusing on strategies for managing errors in distributed systems, handling retries, creating custom error responses, and logging errors in a way that facilitates debugging.

1. Basic Error Handling in Spring Boot

Let’s start with a foundational error handling approach in Spring Boot to set up a baseline.

1.1 Using `@ControllerAdvice` and `@ExceptionHandler`

Spring Boot provides a global exception handler with @ControllerAdvice and @ExceptionHandler. This setup lets us handle exceptions across all controllers in one place.

@RestControllerAdvice
public class GlobalExceptionHandler {

    @ExceptionHandler(ResourceNotFoundException.class)
    public ResponseEntity<ErrorResponse> handleResourceNotFound(ResourceNotFoundException ex) {
        ErrorResponse error = new ErrorResponse("NOT_FOUND", ex.getMessage());
        return new ResponseEntity<>(error, HttpStatus.NOT_FOUND);
    }

    @ExceptionHandler(Exception.class)
    public ResponseEntity<ErrorResponse> handleGeneralException(Exception ex) {
        ErrorResponse error = new ErrorResponse("INTERNAL_SERVER_ERROR", "An unexpected error occurred.");
        return new ResponseEntity<>(error, HttpStatus.INTERNAL_SERVER_ERROR);
    }
}

Here, ErrorResponse is a custom error model:

public class ErrorResponse {
    private String code;
    private String message;

    // Constructors, Getters, and Setters
}

1.2 Returning Consistent Error Responses

Ensuring all exceptions return a consistent error response format (e.g., ErrorResponse) helps clients interpret errors correctly.

2. Advanced Techniques for Error Handling

2.1 Centralized Logging and Tracking with Error IDs

Assigning a unique error ID to each exception helps track specific errors across services. This ID can also be logged alongside exception details for easier debugging.

@ExceptionHandler(Exception.class)
public ResponseEntity<ErrorResponse> handleGeneralException(Exception ex) {
    String errorId = UUID.randomUUID().toString();
    log.error("Error ID: {}, Message: {}", errorId, ex.getMessage(), ex);

    ErrorResponse error = new ErrorResponse("INTERNAL_SERVER_ERROR", 
                                             "An unexpected error occurred. Reference ID: " + errorId);
    return new ResponseEntity<>(error, HttpStatus.INTERNAL_SERVER_ERROR);
}

Clients receive an error response containing errorId, which they can report back to support, linking them directly to the detailed logs.

2.2 Adding Retry Logic for Transient Errors

In distributed systems, transient issues (like network timeouts) can be resolved with a retry. Use Spring’s @Retryable for retry logic on service methods.

Setup

First, add Spring Retry dependency in your pom.xml:

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>

Then, enable Spring Retry with @EnableRetry and annotate methods that need retries.

@EnableRetry
@Service
public class ExternalService {

    @Retryable(
        value = { ResourceAccessException.class },
        maxAttempts = 3,
        backoff = @Backoff(delay = 2000))
    public String callExternalService() throws ResourceAccessException {
        // Code that calls an external service
    }

    @Recover
    public String recover(ResourceAccessException e) {
        log.error("External service call failed after retries.", e);
        return "Fallback response due to error.";
    }
}

This configuration retries the method up to 3 times, with a delay of 2 seconds between attempts. If all attempts fail, the recover method executes as a fallback.

2.3 Using Feign Client with Fallback in Microservices

For error handling in service-to-service calls, Feign provides a declarative way to set up retries and fallbacks.

Feign Configuration

Define a Feign client with fallback support:

@FeignClient(name = "inventory-service", fallback = InventoryServiceFallback.class)
public interface InventoryServiceClient {
    @GetMapping("/api/inventory/{id}")
    InventoryResponse getInventory(@PathVariable("id") Long id);
}

@Component
public class InventoryServiceFallback implements InventoryServiceClient {

    @Override
    public InventoryResponse getInventory(Long id) {
        // Fallback logic, like returning cached data or an error response
        return new InventoryResponse(id, "N/A", "Fallback inventory");
    }
}

This approach ensures that if inventory-service is unavailable, the InventoryServiceFallback kicks in with a predefined response.

3. Error Logging and Observability

3.1 Centralized Logging with ELK Stack

Configure an ELK (Elasticsearch, Logstash, Kibana) stack to consolidate logs from multiple microservices. With a centralized logging system, you can easily trace issues across services and view logs with associated error IDs.

For example, configure log patterns in application.yml:

logging:
  pattern:
    console: "%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n"

3.2 Adding Trace IDs with Spring Cloud Sleuth

In distributed systems, tracing a single transaction across multiple services is critical. Spring Cloud Sleuth provides distributed tracing with unique trace and span IDs.

Add Spring Cloud Sleuth in your dependencies:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>

4. Custom Error Handling for REST APIs

4.1 Creating Custom Exception Classes

Define custom exceptions to provide more specific error handling.

public class InvalidRequestException extends RuntimeException {
    public InvalidRequestException(String message) {
        super(message);
    }
}

4.2 Custom Error Response Structure

Customize error responses by implementing ErrorAttributes for structured and enriched error messages.

@Component
public class CustomErrorAttributes extends DefaultErrorAttributes {

    @Override
    public Map<String, Object> getErrorAttributes(
            WebRequest webRequest, ErrorAttributeOptions options) {
        Map<String, Object> errorAttributes = super.getErrorAttributes(webRequest, options);
        errorAttributes.put("timestamp", LocalDateTime.now());
        errorAttributes.put("customMessage", "An unexpected error occurred");
        return errorAttributes;
    }
}

4.3 API Error Response Standardization with Problem Details (RFC 7807)

Use the Problem Details format for a standardized API error structure. Define an error response model based on RFC 7807:

public class ProblemDetailResponse {
    private String type;
    private String title;
    private int status;
    private String detail;
    private String instance;

    // Constructors, Getters, and Setters
}

Then, return this structured response from the @ControllerAdvice methods to maintain a consistent error structure across all APIs.

@ExceptionHandler(InvalidRequestException.class)
public ResponseEntity<ProblemDetailResponse> handleInvalidRequest(InvalidRequestException ex) {
    ProblemDetailResponse error = new ProblemDetailResponse(
        "https://example.com/errors/invalid-request",
        "Invalid Request",
        HttpStatus.BAD_REQUEST.value(),
        ex.getMessage(),
        UUID.randomUUID().toString()
    );
    return new ResponseEntity<>(error, HttpStatus.BAD_REQUEST);
}

5. Circuit Breakers for Resilience

Integrating a circuit breaker pattern protects your microservice from repeatedly calling a failing service.

Using Resilience4j Circuit Breaker
Add Resilience4j to your dependencies:

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot2</artifactId>
</dependency>

Then, wrap a method with a circuit breaker:

@CircuitBreaker(name = "inventoryService", fallbackMethod = "inventoryFallback")
public InventoryResponse getInventory(Long id) {
    // Call to external service
}

public InventoryResponse inventoryFallback(Long id, Throwable ex) {
    log.warn("Fallback due to error: {}", ex.getMessage());
    return new InventoryResponse(id, "N/A", "Fallback inventory");
}

This setup stops calling getInventory if it fails multiple times, and inventoryFallback returns a safe response instead.

Conclusion

Advanced error handling in Spring Boot microservices includes:

Centralized error handling for consistent responses and simplified debugging.
Retries and circuit breakers for resilient service-to-service calls.
Centralized logging and traceability with tools like ELK and Sleuth.
Custom error formats with Problem Details and structured error responses.
These techniques help ensure your microservices are robust, providing consistent, traceable error responses while preventing cascading failures across services.

How I fixed java.io.IOException: Broken Pipe in Java (Wildfly 10.1)

Bishwa Poudel — Fri, 08 Jan 2021 00:56:19 +0000

Preview

My first path crossed with Broken Pipe issue was when I was gazing at the logs of a software I was working on.

The stack for the software was:

JavaEE 7 with JAX-RS
Java 8
Wildfly 10.1 as application server
Nginx was used as reverse proxy and load balancer.

I could see following traces flooding the log:

java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

I have ignored it a couple of times previously, thinking it hasn't created any havoc in the production server and wasn't quite sure how it was generated and how to handle, or completely prevent it.

What exactly is Broken Pipe?

In simple term, Broken Pipe means that a machine is attempting to read or write data from/to a pipe, while the machine on other end of the pipe has died or been terminated. Now, since the connection is closed, new connection should be established for further transfer of data, or else the data transfer ceases.

How does it occur?

One of the most common cause for this issue is when a client closes an open connection while performing any action (like loading a page, or downloading an attachment). It also happens when forcefully closing certain connections when using a proxy/load balancer like Nginx (like closing the web browser or even simply cancelling a download) or when the connection is slow.

A simple scenario: a browser requesting resource from server, and in response server returns response to browser. What if user closes the browser when server was sending response to browser? The connection between server to browser gets closed unexpectedly. This then gives rise to Broken Pipe and the exception is termed as java.io.IOException: Broken Pipe in Java.

This can happen as well with anything that is interrupting a connection between the client and server including performance issues or even having network intermittency.

Not every Broken Pipe exception is developer's fault

Possible factors giving rise to this exception:

Number of end users
As one of the major cause of Broken Pipe is the behavior of user (unexpectedly closing active browser session before server could complete sending response), increase in number of end user increases chance of having Broken Pipe.
Heavy response load
Heavy response from server takes significant amount of time to get transferred to client side, and this huge time span can be the case for Broken Pipe.
Timeout of server
If the web server couldn't get the response from service in certain time which is equal to timeout value set in server, it closes connection to the client end returning 503: Gateway Timeout and consequently giving rise to Broken Pipe.

Is Broken Pipe Exception a Red Signal?

Well to be fair, it isn't a red signal, because largely it is caused by normal behavior of user, and there are always a chance of services being shutdown due to some failures. But, if a server is running on relatively large number of user requests at a time, then not only Broken Pipe, but any exception seems to pose a problem.

In my case, due to the high network traffic, logs were flooded with broken pipe exception. As, writing to a file (I/O operation) is one of the expensive operation performed by server, imagine server being flooded with Broken Pipe related exception, and the amount of resource server has to put in order to write that exception stacktrace to log file. This had caused server to response slowly and had made it sluggish.

At this point, I realized, YES BROKEN PIPE EXCEPTION IS A RED SIGNAL when scaled to large traffic.

Challenges on handling or removing Broken Pipe

The system was using Wildfly 10.1 as application server, and was written on JavaEE 7. Handling this scenario wasn't a cakewalk for me as:

Replicating this exception in local or QA environment requires all planets to be aligned properly ( just kidding ), but yes it's toooo tough.
Handling exception in Java is easy, as long as the exception is caught inside catch block. The nature of Unhandled exception: java.io.IOException: Broken pipe is such that, it is raised from Wildfly container and gets logged out in stacktrace instead of being trapped in catch block. Now, imagine having to deal with exception that cannot be caught from code. Gosh!!!!
You will never know, which request raised the issue, as, the server was getting high amount of requests, any of them could have been the cause of exception. Adding logs to every rest endpoints along with socket endpoints is not feasible.

Fixing java.io.IOException : Broken Pipe

Finally, after so much of out of context talk, here comes the main section (hope its worth the wait).

Two ways to remove exception from the system are:

Investigate the root cause of the exception, and eliminate it.
Handle the exception gracefully, with proper logging or some action.

Eliminating root causes

Asking user not to close connection unexpectedly
This is impossible to do, for god's sake
Reducing api response load
This is somehow achievable, but in a legacy system, operating on large amount of data, rewriting all the logic so that api responses aren't heavy is also not feasible in all case.
Increasing Timeout of Server
Nginx has a variable named proxy_read_timeout which has default value of 60s, increasing this value will also minimize the chances of Broken Pipe.

Even after eliminating exact root cause, which itself is hard to detect in this case, we can't completely rule out the existence of Broken Pipe. Can we?

Hence, next solution is to handle Broken Pipe gracefully.

Handling java.io.IOException: Broken Pipe

Suppress the log from logger itself If you use log4j as log manager, adding following configuration to log4j.properties will help to get rid of exception flooding the logs due to Broken Pipe.

log4j.logger.org.apache.catalina.connector.ClientAbortException = ERROR, console, cloudAppender
log4j.additivity.org.apache.catalina.connector.ClientAbortException = false

Upgrading Resteasy within Wildfly We were using Resteasy implementation of JAX-RS for implementing REST in Java, and Resteasy v3.0.19 is bundled with Wildfly 10.1. With hope of finding some fixes in the resteasy itself, I started to dig into release notes of Resteasy and found out that after Resteasy-Client v3.1.1, Unhandled Exception: java.io.IOException can be caught from code, unlike container exposing it directly onto logtrace. Hence, upgrading version of Resteasy Client allows us to handle exception through Global Exception Handler (described in later point).

Steps to upgrade Resteasy within Wildfly

Find location of Resteasy distribution jars:
The location of Resteasy distribution jars lies inside:
WILDFLY_HOME/modules/system/layers/base/org/jboss/resteasy/resteasy-jaxrs/main
and the file name is: resteasy-jaxrs-3.0.19.Final.jar
Download the zip file of target version of resteasy (3.1.1 and above) from Resteasy downloads
Unpack the file in temporary directory. It has two other zip files within it:
resteasy-jboss-modules-3.1.1.Final-mavenized.zip &
resteasy-jboss-modules-3.1.1.Final.zip
Unpack both .zip files at:
WILDFLY_HOME/modules/system/layers/base
Update resteasy dependencies' version in pom.xml

<dependency> 
  <groupId>org.jboss.resteasy</groupId> 
  <artifactId>resteasy-jaxrs</artifactId> 
  <version>3.1.1.Final</version>
  <scope>provided</scope>
</dependency>

Use Global Exception Handler I further added Generic Exception Mapper from the java code itself, to catch and handle the Broken Pipe.

My final approach

Though I had gathered multiple techniques to solve the issue, following are the things I actually implemented:

Upgrade Wildfly to v11.0, as it automatically comes with Resteasy v3.1.1 (the version that I exactly wanted) bundled within it.
Handle exception through Global Exception Handler

and with this, the issue of Broken Pipe now only exists in old logs archives 😎😎.

Parting words

It's my first blog, yeah it's a bit long, but I was giving insights into how I actually approached the issue and fixed it. Feel free to provide feedback in the comment section & watch this space for more development blogs in future.

Also, a bit of self promotion at the end 😉. Reach out to me at Twitter & LinkedIn