Why Out-of-Band Health Checks Are the Secret to Hassle-Free Maintenance

Ganapathy Subramanian Ramachandran — Wed, 18 Dec 2024 19:46:15 +0000

If you’ve ever built software that interacts with upstream services, chances are you’ve implemented some kind of health check. Maybe your software is a standalone application, a proxy, or even a service consumed by others. Regardless, health checks are crucial for ensuring your system stays reliable and responsive.

But here's the thing: not all health checks are created equal. In fact, the way you implement them can make or break your ability to maintain your service without impacting users. Let me explain.

The Problem with In-Band Health Checks

The worst (and most common) way to perform health checks is by piggybacking the very same port your service listens on for data. These are called in-band health checks, and they look a lot like this:

A basic HTTP request, like GET /hc.html, to a main service port, which in our examples would be port 80.
A TCP connection check against that primary operating port of a given service.
On the surface, this seems fine—it’s quick, simple, and easy to set up. But here’s the catch: when you’re using the same port for both health checks and client traffic, you’re setting yourself up for trouble.

What Could Go Wrong?

Resource Contention: If your service is experiencing high traffic, health check requests are forced to compete with client requests. This can lead to false alarms about service health.
False Negatives: A minor hitch in the traffic or a load balance can bring your health checks down for the count-even when a service is nominally okay.
Disruptive Maintenance: Performing scheduled updates to take services down? Have fun hoping to do so gracefully whenever your health checks are bound to the same port.
The Smarter Alternative: Out-of-Band Health Checks
So what's the better option? Enter out-of-band health checks.

This approach involves using a separate port to run your health checks. The idea is simple: keep health checks entirely independent of the main service port. For example:

If your service runs on port 80, you’d expose the health check on port 81.
If it’s HTTPS on port 443, the health check could be on port 444.

Why Out-of-Band Checks Are a Game-Changer

Here are the key benefits:

Maintenance Without Downtime: Out-of-band checks allow you to gracefully take a service offline without affecting existing connections. In most cases, you need only fail the health check-simply return a 4xx response-or shut down the health check port-to instruct your load balancers that a particular service is unavailable. Meanwhile, any currently processed client requests can complete without any disruption visible to users.
Improved Accuracy: Since health checks no longer compete with client traffic, you'll have a clearer picture of your service's health. No more false negatives due to spikes in user activity.
Flexibility: You may then introduce custom rules or logic without touching the main service. For instance, you may decide to add extra latency or simulate failures for testing purposes.
Easier Diagnostics: Placing health checks on their own port enables the ability for multiple checks in one spot. For example:

GET /healthcheck/port1 for service A
GET /healthcheck/port2 for service B
It helps much in identifying issues and learning the status of various components.

A Simple Convention for Control Ports

To further simplify things, you may stick to using a fixed offset to assign the health check ports. As an example:

Port 80 → Control port 81
Port 443 → Control port 444
For larger offsets, 8080 and 8443.
This consistency makes it easy to configure and monitor your services.

Wrapping Up

Out-of-band health checks are a small change that pays big dividends. They make your services more resilient, your maintenance less disruptive, and your monitoring more reliable.