Ever had a pod that looked "Running" in your cluster but was actually broken inside, quietly serving errors to every request that hit it? If you've worked with Kubernetes for more than a few weeks, you've probably run into this exact headache. The pod status says everything is fine, but users are seeing errors and your phone won't stop buzzing.
This is exactly the problem health checks were built to solve.
Kubernetes doesn't automatically know if your application is actually working. It only knows if the container process is still alive. Without health checks, Kubernetes is basically flying blind. It assumes that as long as a process hasn't crashed, everything must be okay. That's rarely true. Apps hang, deadlock, run out of memory slowly, or lose connection to a database and just sit there, technically alive but practically useless.
Let's break down what health checks are, the different types, and how to use them well.
What Are Kubernetes Health Checks?
Health checks, which Kubernetes calls probes, are simple instructions that tell Kubernetes how to check whether your application is actually healthy, not just alive.
Think of it like a manager checking in on an employee. Just asking if someone is at their desk isn't enough. A good manager asks if the person is actually able to do their job right now. That's the difference between a process existing and a process being functional.
Kubernetes uses three types of probes, and each one answers a different question. A liveness probe asks whether the app is stuck and needs a restart. A readiness probe asks whether the app is ready to accept traffic right now. A startup probe asks whether the app has finished starting up yet. Understanding these three roles is really the foundation of using health checks well.
Liveness Probes: Should I Restart This?
A liveness probe checks whether a container is stuck in a broken state that only a restart can fix.
This shows up in a few common situations. An app might be deadlocked and never recover on its own. A memory leak might be slowly choking the process until it can't respond anymore. An infinite loop bug might leave the app technically running but completely unresponsive.
When a liveness probe fails repeatedly, Kubernetes kills the container and starts a fresh one. That restart is the entire fix it offers.
The key rule here is to keep liveness checks focused only on the app itself. Don't use them to check external things like databases or APIs. That's one of the most common mistakes teams make, and we'll get to it shortly.
Readiness Probes: Should I Send Traffic Here?
This is the probe most people confuse with liveness, but it solves a different problem entirely.
A readiness probe tells Kubernetes whether a pod should currently receive traffic. If it fails, Kubernetes doesn't restart the pod. It simply stops sending traffic to it until the probe passes again.
This matters most during startup, when the app is loading configs, warming caches, or connecting to a database. It also matters during temporary overload, when a pod is too busy to handle more requests, and during dependency issues, like a lost connection to a downstream service.
Imagine an API that needs to connect to a database before it can serve requests. Without a readiness probe, Kubernetes might send traffic to it before that connection is ready, resulting in failed requests during every deployment or restart. With a readiness probe checking the database connection, the pod simply won't receive traffic until it's truly ready, and users never notice a thing.
Startup Probes: Give It Time to Wake Up
Some applications are just slow to start. Older systems especially can take a few minutes before they're truly ready.
The problem is that if Kubernetes checks too early, it might assume the app is broken and restart it before it even finishes booting. This creates a frustrating loop where the app never gets a real chance to start.
Startup probes solve this by pausing liveness and readiness checks until the app has fully started. Once the startup check passes even once, Kubernetes switches over to its normal monitoring routine.
How Kubernetes Checks Health
There are three simple ways probes can check an application. One method sends a request to a specific endpoint and checks the response. Another checks whether a port is open and accepting connections. A third runs a small command inside the container and checks whether it succeeds.
Which method you use really depends on your app. Web services typically use the first approach, databases or queues often use the second, and custom tools sometimes rely on the third.
Best Practices for Health Checks
It helps to build a dedicated health endpoint rather than reusing a business logic endpoint for this purpose. A lightweight, dedicated path that responds quickly and doesn't depend on heavy processing works best.
Keep liveness checks simple and focused only on the app's own internal state. If a liveness check also tests a database and that database goes down, Kubernetes will restart every pod, which does nothing to fix the database and just adds chaos to an existing outage.
Let readiness probes handle dependency checks instead. This is the right place to check things like database or API connectivity. If a dependency is down, the pod gets marked as not ready, and traffic simply routes to healthy pods.
Tuning the timing carefully makes a real difference too. A few settings control how probes behave, including how long to wait before the first check, how often to repeat it, how long to wait for a response before it counts as failed, how many failures in a row trigger action, and how many successes in a row mark it healthy again. Getting these numbers right, based on real app behavior rather than guesses, prevents both false alarms and slow detection of genuine problems.
It also helps to match timeouts to reality. If your health check sometimes takes a couple of seconds to respond under load, don't set an unrealistically short timeout, or you'll end up with false failures and unnecessary restarts.
Common Mistakes to Avoid
One frequent mistake is using the same check for both liveness and readiness, even though they serve different purposes. Copying one config for both often causes unnecessary restarts.
Another common issue is checking external dependencies inside a liveness check. This turns a downstream outage into a full application outage, since Kubernetes ends up restarting pods that were never actually broken.
Starting checks too early for slow booting apps is another trap. This creates restart loops before the app has a real chance to boot, and a startup check solves this far better than just extending the initial wait time.
Ignoring probe failures in logs is also common. These failures are visible if you look, but many teams only notice them once an incident has already happened.
Being too aggressive with failure limits causes trouble as well. Checking too frequently with very low tolerance means a single network blip can trigger an unnecessary restart.
Finally, some teams skip health checks entirely to keep things simple. This almost always backfires the first time something actually goes wrong in production.
Actionable Tips You Can Apply Today
- Check pod events whenever you're debugging strange traffic behavior, since probe failures usually show up clearly there.
- Add simple logging inside your health check endpoints so failures are easy to trace later.
- Test your health endpoints manually before deploying, especially after changing app logic.
- Use separate endpoints for liveness and readiness so you can tune each one independently.
- Start with conservative, forgiving settings and tighten them only after observing real behavior over time.
- Always add a startup check for apps with unpredictable boot times, rather than just extending the initial wait time indefinitely.
Conclusion
Health checks are one of those Kubernetes features that seem simple on the surface but have real consequences when misconfigured. Getting them right means smoother deployments, fewer late night alerts, and traffic that only ever reaches pods that can actually handle it. Getting them wrong means restart loops, failed rollouts, and outages that could have been avoided with a little more care.
Once you understand the distinct roles of liveness, readiness, and startup checks, configuring them well becomes second nature. Start simple, observe real behavior, and tune from there.
Key Takeaways
- Liveness probes answer whether a pod should be restarted, so keep them focused only on the app's own state.
- Readiness probes answer whether a pod should receive traffic, making them the right place to check dependencies.
- Startup probes protect slow starting apps from being killed before they finish booting.
- External dependencies should never be checked inside a liveness check.
- Timing settings should be tuned based on real observed behavior, not assumptions.
- Dedicated, lightweight health endpoints make debugging and tuning far easier for everyone on the team.
FAQ
1. What's the main difference between liveness and readiness probes?
Liveness probes decide if a pod should be restarted. Readiness probes decide if a pod should receive traffic. A pod can be alive but not ready, especially during startup.
2. What happens when a liveness probe fails?
Kubernetes restarts the container after repeated failures.
3. What happens when a readiness probe fails?
The pod stops receiving new traffic, but it isn't restarted.
4. Do I need all three probe types for every application?
Not necessarily. Simple, fast starting apps might only need liveness and readiness checks. Startup checks mainly help slow booting applications.
5. Can I use the same check for liveness and readiness?
You can, but it's not recommended, since the two checks serve different purposes.
6. Should my liveness check test database connectivity?
No. That belongs in the readiness check. A liveness check should only confirm the app itself is functioning.
7. What is a startup probe actually for?
It gives slow starting apps extra time to boot before liveness and readiness checks begin, preventing premature restarts.
8. How do I debug a failing probe?
Check pod events and application logs around the time the failure occurred.
9. Can probes cause a restart loop?
Yes, especially if checks start too early for an app that needs more time to boot.
10. What's the difference between the three check methods?
One sends a request to an endpoint, one checks if a port accepts connections, and one runs a command inside the container and checks if it succeeds.
11. Are health checks required in Kubernetes?
No, they're optional, but skipping them means Kubernetes has no real way to know if your app is actually working.
12. How often should checks run?
It depends on the app, but a common starting point is every ten seconds, adjusted based on how quickly you need to catch problems.
13. What does the failure threshold setting control?
It's the number of consecutive failures needed before Kubernetes takes action. Too low causes false alarms, and too high delays detection of real problems.
14. Do health checks slow down my application?
The impact is minimal, as long as your health check endpoints stay lightweight and avoid heavy logic.
15. What's the biggest mistake teams make with health checks?
Checking external dependencies inside the liveness probe, which turns a downstream outage into a full application outage.
See the Real Cost of Unhealthy Pods With EcoScale
Reliability starts with visibility. EcoScale makes sure your teams actually have it.
You can't fix flaky liveness checks, wasted restarts, or pods sitting idle while marked ready if nobody can see where the resource waste is actually coming from. EcoScale gives every team a clear, real-time view of Kubernetes usage and cost, broken down by namespace and workload, so the impact of misconfigured probes stops being invisible and becomes something every team can see, understand, and fix.
If your organization is still dealing with unnecessary restarts, wasted compute, or unclear ownership of Kubernetes reliability, EcoScale can help you build the visibility that makes real fixes possible in the first place.
Book a Free EcoScale Demo: https://ecoscale.dev/#booking
Learn More: https://ecoscale.dev
Top comments (0)