Vicente Reyes wrote up a deploy debugging story this week. Ten small breakages stacked in series, each one hiding the next. Good piece. Go read it.
The last act is the one I want to pick up.
Two minutes after his first fully automated deploy, the site returned a 502. Nothing had crashed. The old container was gone, the new one hadn't finished booting, and he happened to refresh inside a window about one second wide.
That window is in almost every Docker Compose deploy I have seen. It is in mine.
The interesting part isn't the window. It's that the obvious fix only closes half of it, and the other half fails much more quietly.
The proxy has no idea
A static Traefik route looks like this:
http:
services:
api:
loadBalancer:
servers:
- url: http://api:3000
That config says: send traffic to api:3000. It does not say: send traffic to api:3000 when something is actually listening there. Traefik has no opinion on the matter. It resolves the name, opens a connection, reports whatever comes back.
So every time docker compose up -d stops the old container and starts the new one, there is a stretch where the proxy is confidently routing at a process that is still running migrations. Nobody hits the app during that stretch, you never find out.
The fix everyone reaches for is an active health check.
http:
services:
api:
loadBalancer:
servers:
- url: http://api:3000
healthCheck:
path: /healthz
hostname: api.example.com
interval: "5s"
timeout: "3s"
Correct, and it's where most write-ups stop.
The hostname line is not optional
Leave hostname out and Traefik sends the check with Host: api, the internal Docker service name. Anything with a host allowlist in front of it will reject that. Django gives you a DisallowedHost 400. Express with a host check does the same. So does NestJS behind most gateway setups.
From Traefik's side a 400 is a failed check, and a container that fails every check is permanently unhealthy.
So you spend an afternoon debugging why your app is down. Your app is fine. The health check is asking the question in a form the app is contractually obliged to refuse.
Set hostname to the real public domain. Then the check looks like real traffic, which is the whole point of it.
Liveness and readiness are different questions
The advice attached to that config is usually "keep /healthz trivial". No database, no auth, just return 200. That advice is right, and for a good reason: if your health endpoint touches a business dependency, a permissions change or one slow query quietly pulls the container out of rotation.
But a trivial endpoint answers a trivial question. It answers is this process alive. It does not answer can this process serve a request.
Those two come apart constantly. A container that boots cleanly but can't reach its database will return 200 from a trivial check all day. Pool empty. Queue consumer never attached. The proxy sees green, sends real traffic, and instead of one second of clean 502s you get sustained 500s from a container that is reporting healthy.
So split them. Liveness stays trivial and wires to your restart policy. Readiness checks what the container needs to do its job and wires to the load balancer. Collapse the two and you pick one failure mode to catch and get blindsided by the other.
The part nobody writes down
This is the bit I actually sat down to write.
The natural readiness check for a service that depends on Postgres is a SELECT 1. Cheap, honest, and it's what I'd reach for too.
Think about what it proves. It proves the pool can hand you a connection. It says nothing about whether the pool is nearly empty.
Under load those diverge, and the divergence is ugly. If readiness borrows from the same pool the application serves from, then when that pool is close to exhausted the check either succeeds by taking the last free connection, which is the moment you least wanted a health probe competing with real requests, or it fails outright because there was nothing left to take.
The second one is what hurts.
A container fails readiness. The load balancer pulls it. Its traffic redistributes to its neighbours. Those neighbours now carry more load, their pools saturate, they fail readiness too, and out they go.
You have built a check whose entire job is to protect the system, and under exactly the conditions it was meant to protect against, it removes the system.
Three cheap ways out, pick one:
Give readiness a dedicated connection outside the application pool. Then a saturated pool is a slow app rather than an unhealthy one.
Or keep it on the shared pool and be honest about what it asserts, which is "Postgres is reachable from this container". Fine thing to check, as long as nobody downstream believes it means "has capacity".
Either way, never let a single failed probe evict. Require consecutive failures over a window so a momentary squeeze can't cascade.
The general principle is that a readiness check consuming the same scarce resource it measures will always give you its worst answer at the worst possible moment.
Readiness is a dependency graph you're signing up for
Everything you put in a readiness check becomes something that can take you out of rotation.
Put a downstream HTTP call in there and you have coupled your availability to somebody else's. They have a bad ten minutes, every one of your containers marks itself unready simultaneously, and now you have zero healthy instances of a service that was working perfectly well.
So the scoping rule is narrow. Check only what this container needs to serve its own traffic.
Vicente landed on exactly this for his stack. Postgres in, Redis and Celery out, on the grounds that background jobs degrading is not the same thing as the web tier being unable to answer. Redis going down should page someone. It should not empty the load balancer.
That's a design decision, not a config line, and it's worth the ten minutes.
Start-first doesn't get you to zero either
Say you fix all of that. The window shrinks from "boot plus migrations plus static assets" down to "process fork time".
It shrinks. It does not close. docker compose up -d stops the old container before the new one is ready, and no health check changes that ordering.
The usual next step is a start-first rollout. Bring up the new container, wait for it to pass readiness, kill the old one. The docker-rollout plugin does this, Swarm and Kubernetes do it natively.
The bit that gets skipped: start-first closes the window where nothing is listening and opens a different one, where the old container takes SIGTERM with requests still in flight.
Two things have to be true for that second window to be harmless. The proxy has to notice the old server left and stop sending it new work before the container dies, and Traefik's config reload is not instant. And the application has to be allowed to finish what it is already holding. In gunicorn that's graceful-timeout. In Node it's whatever you wired into your SIGTERM handler, and if you wired nothing, the answer is that it doesn't.
A rollout without a drain window doesn't eliminate dropped requests. It moves them from the front of the deploy to the back, where they are harder to spot because the deploy reports success.
If your app is anywhere near money that distinction matters a lot. A request dropped at the front is a failed connection and the client retries. A request dropped at the back was accepted, was being processed, and then wasn't.
What to actually do
The answer is different at each size, so:
Single box, Compose, low traffic. Split liveness and readiness, scope readiness to your own datastore, set hostname, live with a health-gated couple of seconds. Completely reasonable place to stop. Reaching for an orchestrator here costs you more than the window does.
Multiple instances behind a proxy. All of the above, plus a failure threshold so one bad probe can't evict, plus readiness on its own connection. This is where the cascade becomes possible, so this is where it's worth paying for.
Deploys that cost you money when they drop requests. Start-first rollout with an explicit drain. Proxy deregistration before SIGTERM, graceful timeout longer than your slowest normal request.
The tools matter less than the question.
Every health check is an assertion, and the failure is always the same one. The assertion is narrower than what everyone downstream believes it to be. Somebody reads a trivial 200 as "can serve". Somebody reads a SELECT 1 as "has capacity". Somebody reads a green deploy as "nobody got a 500".
Write down what your check actually proves. Then go and find out who is relying on it to prove more than that.
Thanks to Vicente G. Reyes for the original write-up and for a good argument in the comments. Most of this is that argument.
I'm Arun, CTO and co-founder at Atoa. We build open banking payments for the UK. I write about the messy parts of running engineering systems. @mickyarun.
Top comments (0)