Monitoring the health of Spring Boot applications with custom health indicators

Disclaimer

This article represents my perspective on this solution, thus any suggestions, fixes, or discussions will be highly appreciated.

The short story

In today's complex software landscape, understanding the health status of an application has never been more crucial. An application's "health" extends beyond just uptime. It involves the ability of the application to interact effectively with the external sources it relies on, such as databases, APIs, or caches.

We use Spring Boot applications extensively, and these depend significantly on various external data sources such as bespoke APIs, third-party APIs, and internal caches. The primary objective is to ensure the robust health of these applications within a Kubernetes deployment.

Each application has its own unique health state, determined by the status of the external sources it uses. For instance, if your application communicates with a third-party service or a database, its health status is not only reliant on its internal functionality but also on the availability and responsiveness of these external systems.

If an essential external source experiences downtime, it could significantly affect our application's performance and business logic, even if the application itself is running perfectly. Hence, it's critical to consider these external factors when determining the health status of your application.

Health checks offer a solution to this challenge. They allow you to monitor and assess the state of the systems your application relies on. If an external source is important for your application, you need to include it in your health checks.

For example, if your application relies heavily on a particular API, you should implement a health check that ensures the API is up and running. This way, you'll be alerted if there's a problem, and you can address it promptly to prevent any severe consequences for your application.

Spring Boot's Health Indicator

Spring Boot provides a great feature that helps with this – custom health indicators. The HealthIndicator interface provides a method, health(), that returns a Health response. You can define the health rules according to your application's requirements, which can range from checking database connectivity to validating the status of an external service.

With Spring Boot, you can customize these health indicators according to your specific needs, ensuring the accuracy of your application's health status. For example, if your application uses an external API and a cache system, you could create two separate health indicators – one for the API and one for the cache. If either system encounters a problem, the relevant health indicator will report it, enabling you to take prompt action.

The implementation

Let's configure two kinds of health indicators:

1. for an external API
2. for an internal cache

Health indicator for an external API

Our custom ApiHealthIndicator interface creates a contract for all external APIs we would like to monitor by providing a doHealthCheck() method. The health status will we be created by default methods of this interface: a healthy state and a non-healthy state.

You can add additional details to the down health state like custom runtime exception, a detailed message etc.

import org.springframework.boot.actuate.health.Health;
import reactor.core.publisher.Mono;

import static org.apache.commons.lang3.BooleanUtils.isFalse;
import static org.apache.commons.lang3.BooleanUtils.isTrue;

public interface ApiHealthIndicator {

    Mono<Health> doHealthCheck();

    default Health createHealth(Boolean status, String serviceName) {
        return isTrue(status)
                ? new Health.Builder().up().build()
                : createHealthDownStatus(status, serviceName);
    }

    default Health createHealthDownStatus(String serviceName) {
        var exception = new ServiceUnavailableException(String.format("The %s isn't available", serviceName));
        return new Health.Builder()
                .down(exception)
                .build();
    }

}

The component ServiceStateHealthIndicator effectively performs the required logic: it integrates a service bean that retrieves the current status from an external API, indicating whether the API is available or unavailable (true/false). When a monitoring resource, such as Kubernetes, sends a health check request to the application's HealthEndpoint through the actuator, the getHealth() method gets triggered. This method furnishes statuses from all HealthContributor instances, which includes our custom ReactiveHealthIndicator.

import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.ReactiveHealthIndicator;
import org.springframework.stereotype.Component;
import reactor.core.publisher.Mono;

@Component
public class ServiceStateHealthIndicator implements ReactiveHealthIndicator, ApiHealthIndicator {

    private static final String SERVICE_NAME = "external-api";

    private final ApiService service;

    public ServiceStateHealthIndicator(ApiService service) {
        this.service = service;
    }

    @Override
    public Mono<Health> health() {
        return doHealthCheck()
                .onErrorResume(exception -> Mono.just(createHealthDownStatus(false, SERVICE_NAME)));
    }

    @Override
    public Mono<Health> doHealthCheck() {
        return service.getStatus()
                .map(status -> createHealth(status, SERVICE_NAME));
    }

}

And of course our custom exception to explain which status it reflects:

public class ServiceUnavailableException extends RuntimeException {

    public ServiceUnavailableException(String message) {
        super(message);
    }

}

Health indicator for an internal cache

Suppose one of our application's health prerequisites is the presence of an internal cache. If the cache is empty, we'd prefer to prevent the application from starting up altogether. To facilitate this, we can tailor the readiness state using Spring Boot's ReadinessState.class to control the availability state.

Let's assume we have an interface, CacheService, that provides an isNotEmpty() method implemented in each class related to our internal cache. This method delivers a straightforward boolean result, enabling us to verify if each of our internal caches is in a valid state.

By injecting a map containing all cache service beans, we can cycle through each of our caches. If even a single cache is empty, the readiness state prevents any traffic from reaching the application.

We can extend the AvailabilityStateHealthIndicator in Spring Boot to create a mapping of statuses and adjust the getState() method. This method yields an instance of ReadinessState based on our caches' state.

The ReadinessStateHealthIndicator is charged with providing this status when an external monitoring tool, such as Kubernetes, dispatches an HTTP request to our application's readiness endpoint.

import org.springframework.boot.actuate.availability.AvailabilityStateHealthIndicator;
import org.springframework.boot.actuate.health.Status;
import org.springframework.boot.availability.ApplicationAvailability;
import org.springframework.boot.availability.AvailabilityState;
import org.springframework.boot.availability.ReadinessState;
import org.springframework.stereotype.Component;

import java.util.Map;

@Component
public class CacheStateHealthIndicator extends AvailabilityStateHealthIndicator {

    private final Map<String, CacheService> cacheServices;

    public CacheStateHealthIndicator(Map<String, CacheService> cacheServices, ApplicationAvailability availability) {
        super(availability, ReadinessState.class, statusMappings -> {
            statusMappings.add(ReadinessState.ACCEPTING_TRAFFIC, Status.UP);
            statusMappings.add(ReadinessState.REFUSING_TRAFFIC, Status.OUT_OF_SERVICE);
        });
        this.cacheServices = cacheServices;
    }

    @Override
    protected AvailabilityState getState(ApplicationAvailability applicationAvailability) {
        boolean isEmpty = cacheServices.values()
                .stream()
                .map(CacheService::isNotEmpty)
                .toList()
                .contains(false);
        if (isEmpty) return ReadinessState.REFUSING_TRAFFIC;
        return ReadinessState.ACCEPTING_TRAFFIC;
    }

}

Summary

In conclusion, Spring Boot offers a comprehensive and customizable system for monitoring the health of your applications. Its HealthIndicator interface allows you to consider all critical factors, including essential external sources, in determining your application's health status. This functionality helps to prevent minor issues from escalating into significant problems, enhancing the reliability of your applications.

In the next article I'll show how can we use these customized health indicators for liveness and readiness Kubernetes probes within application.properties file.

Resources

1. HealthIndicator
2. Spring Boot health indicators
3. ApplicationAvailability
4. AvailabilityStateHealthIndicator

---

Finding my articles helpful? You could give me a caffeine boost to keep them coming! Your coffee donation will keep my keyboard clacking and my ideas brewing. But remember, it's completely optional. Stay tuned, stay informed, and perhaps, keep the coffee flowing!