Vicente Reyes wrote up a deploy debugging story this week. Ten small breakages stacked in series, each one hiding the next. Good piece. Go read it.
The last act is the one I want to pick up.
Two minutes after his first fully automated deploy, the site returned a 502. Nothing had crashed. The old container was gone, the new one hadn't finished booting, and he happened to refresh inside a window about one second wide.
That window is in almost every Docker Compose deploy I have seen. It is in mine.
The interesting part isn't the window. It's that the obvious fix only closes half of it, and the other half fails much more quietly.
The proxy has no idea
A static Traefik route looks like this:
http:
services:
api:
loadBalancer:
servers:
- url: http://api:3000
That config says: send traffic to api:3000. It does not say: send traffic to api:3000 when something is actually listening there. Traefik has no opinion on the matter. It resolves the name, opens a connection, reports whatever comes back.
So every time docker compose up -d stops the old container and starts the new one, there is a stretch where the proxy is confidently routing at a process that is still running migrations. Nobody hits the app during that stretch, you never find out.
The fix everyone reaches for is an active health check.
http:
services:
api:
loadBalancer:
servers:
- url: http://api:3000
healthCheck:
path: /healthz
hostname: api.example.com
interval: "5s"
timeout: "3s"
Correct, and it's where most write-ups stop.
The hostname line is not optional
Leave hostname out and Traefik sends the check with Host: api, the internal Docker service name. Anything with a host allowlist in front of it will reject that. Django gives you a DisallowedHost 400. Express with a host check does the same. So does NestJS behind most gateway setups.
From Traefik's side a 400 is a failed check, and a container that fails every check is permanently unhealthy.
So you spend an afternoon debugging why your app is down. Your app is fine. The health check is asking the question in a form the app is contractually obliged to refuse.
Set hostname to the real public domain. Then the check looks like real traffic, which is the whole point of it.
Liveness and readiness are different questions
The advice attached to that config is usually "keep /healthz trivial". No database, no auth, just return 200. That advice is right, and for a good reason: if your health endpoint touches a business dependency, a permissions change or one slow query quietly pulls the container out of rotation.
But a trivial endpoint answers a trivial question. It answers is this process alive. It does not answer can this process serve a request.
Those two come apart constantly. A container that boots cleanly but can't reach its database will return 200 from a trivial check all day. Pool empty. Queue consumer never attached. The proxy sees green, sends real traffic, and instead of one second of clean 502s you get sustained 500s from a container that is reporting healthy.
So split them. Liveness stays trivial and wires to your restart policy. Readiness checks what the container needs to do its job and wires to the load balancer. Collapse the two and you pick one failure mode to catch and get blindsided by the other.
The part nobody writes down
This is the bit I actually sat down to write.
The natural readiness check for a service that depends on Postgres is a SELECT 1. Cheap, honest, and it's what I'd reach for too.
Think about what it proves. It proves the pool can hand you a connection. It says nothing about whether the pool is nearly empty.
Under load those diverge, and the divergence is ugly. If readiness borrows from the same pool the application serves from, then when that pool is close to exhausted the check either succeeds by taking the last free connection, which is the moment you least wanted a health probe competing with real requests, or it fails outright because there was nothing left to take.
The second one is what hurts.
A container fails readiness. The load balancer pulls it. Its traffic redistributes to its neighbours. Those neighbours now carry more load, their pools saturate, they fail readiness too, and out they go.
You have built a check whose entire job is to protect the system, and under exactly the conditions it was meant to protect against, it removes the system.
Three cheap ways out, pick one:
Give readiness a dedicated connection outside the application pool. Then a saturated pool is a slow app rather than an unhealthy one.
Or keep it on the shared pool and be honest about what it asserts, which is "Postgres is reachable from this container". Fine thing to check, as long as nobody downstream believes it means "has capacity".
Either way, never let a single failed probe evict. Require consecutive failures over a window so a momentary squeeze can't cascade.
The general principle is that a readiness check consuming the same scarce resource it measures will always give you its worst answer at the worst possible moment.
Readiness is a dependency graph you're signing up for
Everything you put in a readiness check becomes something that can take you out of rotation.
Put a downstream HTTP call in there and you have coupled your availability to somebody else's. They have a bad ten minutes, every one of your containers marks itself unready simultaneously, and now you have zero healthy instances of a service that was working perfectly well.
So the scoping rule is narrow. Check only what this container needs to serve its own traffic.
Vicente landed on exactly this for his stack. Postgres in, Redis and Celery out, on the grounds that background jobs degrading is not the same thing as the web tier being unable to answer. Redis going down should page someone. It should not empty the load balancer.
That's a design decision, not a config line, and it's worth the ten minutes.
Start-first doesn't get you to zero either
Say you fix all of that. The window shrinks from "boot plus migrations plus static assets" down to "process fork time".
It shrinks. It does not close. docker compose up -d stops the old container before the new one is ready, and no health check changes that ordering.
The usual next step is a start-first rollout. Bring up the new container, wait for it to pass readiness, kill the old one. The docker-rollout plugin does this, Swarm and Kubernetes do it natively.
The bit that gets skipped: start-first closes the window where nothing is listening and opens a different one, where the old container takes SIGTERM with requests still in flight.
Two things have to be true for that second window to be harmless. The proxy has to notice the old server left and stop sending it new work before the container dies, and Traefik's config reload is not instant. And the application has to be allowed to finish what it is already holding. In gunicorn that's graceful-timeout. In Node it's whatever you wired into your SIGTERM handler, and if you wired nothing, the answer is that it doesn't.
A rollout without a drain window doesn't eliminate dropped requests. It moves them from the front of the deploy to the back, where they are harder to spot because the deploy reports success.
If your app is anywhere near money that distinction matters a lot. A request dropped at the front is a failed connection and the client retries. A request dropped at the back was accepted, was being processed, and then wasn't.
What to actually do
The answer is different at each size, so:
Single box, Compose, low traffic. Split liveness and readiness, scope readiness to your own datastore, set hostname, live with a health-gated couple of seconds. Completely reasonable place to stop. Reaching for an orchestrator here costs you more than the window does.
Multiple instances behind a proxy. All of the above, plus a failure threshold so one bad probe can't evict, plus readiness on its own connection. This is where the cascade becomes possible, so this is where it's worth paying for.
Deploys that cost you money when they drop requests. Start-first rollout with an explicit drain. Proxy deregistration before SIGTERM, graceful timeout longer than your slowest normal request.
The tools matter less than the question.
Every health check is an assertion, and the failure is always the same one. The assertion is narrower than what everyone downstream believes it to be. Somebody reads a trivial 200 as "can serve". Somebody reads a SELECT 1 as "has capacity". Somebody reads a green deploy as "nobody got a 500".
Write down what your check actually proves. Then go and find out who is relying on it to prove more than that.
Thanks to Vicente G. Reyes for the original write-up and for a good argument in the comments. Most of this is that argument.
I'm Arun, CTO and co-founder at Atoa. We build open banking payments for the UK. I write about the messy parts of running engineering systems. @mickyarun.
Top comments (17)
The composite is the honest part, so I would not try to name it precisely. I would date it.
/healthz-or-whatever-it-is-called settles one thing: "this summary was accurate as of the check that last ran." That sentence is true no matter how many parts you fold in, and it can be kept true by the check itself, not by the person maintaining the repo. The parts list is what degrades, so the parts list has to live where the reader is at 3am, which is the alert body, not the runbook nobody opens.
The part of your reply I keep turning over is the reader. You are right that the rename is a judgement about the reader, and that is why it is not fully in your hands: the person who reads green at 03:14 has never seen the check, the runbook, or this post. They read the name as the answer to the question they already had. Which means the name is not really a name. It is a claim with a population, and the population is whatever the reader assumes it covers. /healthz says "one process is listening" and gets read as "we can take payments". Nobody miscounted. The sentence swapped the population.
That is why I would watch for the moment of renaming-back you named, and treat it as the useful signal it is: not a preference about filenames, but somebody wanting the check to cover a claim it does not make. The question I would put to them is not "why the old name" but "what do you want green to mean when the phone rings". If they can answer that in one sentence, you have the real check and the name follows. If they cannot, no filename saves the incident.
"A claim with a population, and the population is whatever the reader assumes it covers" is the sentence. Nobody miscounted. The sentence swapped the population. I had this filed as a naming problem and it is not one, because a name cannot carry a population at all.
The date works for a reason worth naming, because it is the test for everything else. "Accurate as of the check that last ran" stays true with nobody maintaining it. The check writes the timestamp. There is no edit anyone can forget.
Run that test on the parts list and it fails, including in the alert body. A written parts list drifts the first time somebody adds a dependency to the check and not to the string, and nothing breaks when they do. Same failure class as the check itself. Moving it into the alert body does not fix that, it moves it somewhere more people see it being wrong.
The version that passes your test is a parts list generated from whatever the check iterates, so there is one list and both the check and the message read it. If it has to be typed by hand I would rather it were not there, because a stale parts list is worse than none: it reads as the population and it is out of date.
Where I think the argument stops is the 03:14 reader you put at the centre of it. The parts list in the alert body is not read at 03:14 either. They read the colour. The timestamp survives that and the parts list does not, and not because of where it lives. A stale date is visible without parsing anything. A missing dependency is not.
The parts list is where I was wrong, and you found the reason: I moved it, I did not generate it.
Anything a human types into the message drifts, and a stale population is worse than no population, because it reads as the answer. One object, iterated by the check and rendered into the message, is the only version of that list I would defend. If it has to be typed by hand, leave it out.
What I would keep is a smaller claim than the one you are testing. Not "the list is accurate" but "the list is the thing that ran". A generated list cannot be more complete than the check, which is fine. It cannot be wrong about what the check did, which is the point. Same shape as your timestamp: the property comes from the fact that nobody writes it.
And on the 03:14 reader, you are right and I was arguing the wrong defence. They read the colour. No list in any body reaches them. That is why the rename is load-bearing and the parts list is not borrowed by them at all. The reader who gets the list is the one at 09:00 opening the incident, and their question is not "what is this" but "what did it cover". Different reader, different artefact. My mistake was dressing the 09:00 list up as the 03:14 fix.
Your smaller claim is better than the one I was testing, and I'll take it: the list can't be wrong about what the check did, and that's a provenance claim, not a coverage one. Provenance is the only thing a check's own body can honestly carry.
Where it still lets someone through: a generated list is unfalsifiable in the direction that hurts. The check iterates three of seven dependencies, the list prints three, every word of it is true, and nothing in the message knows seven was the number. Correct about what ran, silent about what didn't. So the generated list buys you "this is real" and not "this is all" — and the second one is what the 03:14 reader would want, if they were reading.
Which puts a count from somewhere else back into the message: reached against intended, where intended comes from the service registry or the dependency manifest rather than from the check. Same shape as your timestamp, one layer out. The denominator has to be written by something that isn't the check.
And yes on 03:14. That's the argument for the rename being the whole fix rather than part of it. If colour and name are the only two fields that reach a person mid-incident, every other field in that body is written for the post-mortem on Tuesday morning. Worth saying out loud, because most health check bodies are written as though someone reads them while paged.
You are right that the denominator cannot come from the check. I think the fraction you build out of it fails the same test one layer further out.
3 of 7 declared, rendered as a number, is a ratio, and a ratio invites a threshold. Whoever decides that four unexercised is amber and three is red has made a policy call inside an artefact that is supposed to be a measurement. Same coat as an invented cost line: a number the reader treats as measured because it printed.
What I would keep from your move is the direction of falsifiability. Your numerator is checkable against the check's own body. If the message says three and the code iterated four, someone can catch it. Your denominator is checkable against nothing except the manifest it came from, and the manifest describes what was declared, not what was required. So the ratio is reached over declared, and the sentence a reader hears is reached over required. A wrong numerator is a bug. A wrong denominator is a belief.
Which is why I would not render the fraction at all. Render the gap as its own field, and make it non-aggregating: "four declared dependencies not exercised by this run". That is either zero or it is not, it can never turn green, and it needs no threshold to be read. A count of absences survives drift better than a proportion of them, because nothing rounds a gap up.
The ratio goes. You're right about why, and it is the cleanest version of the argument so far: a threshold is a policy call wearing a measurement's clothes, and once the number prints, somebody tunes it. I was going to put a fraction in a field and then act surprised when it grew an amber band.
"A wrong numerator is a bug. A wrong denominator is a belief." That is the line I will be repeating.
What the gap field does not buy, though, is escape from the denominator. "Four declared dependencies not exercised by this run" is reached-over-declared with the division left out. The belief is still in there. It just cannot be rounded. So the win is smaller than survives-drift-better. The win is that nobody can tune it, and tuning was the failure mode.
Where I think it breaks: that field will be non-zero forever on a real service. Some dependencies are declared and genuinely do not belong in this check's claim. A field that never reaches zero gets learned around in about three weeks, same as an amber band set too wide. Permanently non-zero and ignored is the same outcome as a threshold, by a different route.
The only fix I can see puts policy straight back in. The absence has to be acknowledgeable per dependency, with a name on it. Not "four unexercised" but "four unexercised, three signed off by someone, one nobody has looked at". Policy does not leave the artefact. It stops being a threshold and becomes attributable. A band hides whose judgement it was. A waiver has an author.
Then the part I have no answer for: nobody keeps those waivers current. It is your parts list again, one layer out, with the same property that nothing breaks when it goes stale. The only difference I would claim is that a stale waiver has a name attached, so there is someone to ask. That is weaker than generation. It is the best I have where the fact in question is a human decision and cannot be derived from anything the code iterates.
Permanently non-zero and ignored is the same outcome as a threshold. I accept that, and it means the gap field as I wrote it was half a fix.
The waiver is the one thing in this thread that is not a measurement, and I think that is the answer: it should not be kept current. It should expire.
Everything else here fails by drift. The parts list, the denominator, the amber band: they go stale while still reading as true, and nothing breaks when they do. A waiver can fail the other way if you let the check compute its term against today. Then a forgotten waiver does not stay quietly acknowledged. It decays back into an unacknowledged gap, which is the safe state, and it does that without anyone remembering anything. Same property as your timestamp: the sentence is true because nobody writes it.
The field then reads: four unexercised, three acknowledged and current, one acknowledged and expired. Nothing to tune, nothing to round, and the expired row carries the name of whoever signed it.
Where it stops, and I think this is where your "nobody keeps them current" actually lands: a term long enough to be unannoying is long enough to be renewed by reflex. Ninety days is a click, and after the click the name means nothing. Seven days is a chore, and the check gets muted. There is no term that fixes that, because what you are asking for is a judgement, and judgements decay. The term does not stop the decay. It makes the decay show up as a date instead of as silence, which is more than any threshold manages.
Expiry is the move, and the reason is sharper than surviving drift. Every other field in this thread fails toward still reads true. A waiver with a computed term fails toward unacknowledged, which is the state you'd pick if you had to pick blind. Direction of failure, not rate of it.
On the term, you've already conceded it, and I'd go further: this mechanism has field data. Regulated payments has run it for years and calls it an attestation. A compensating control with an owner and a review date. Reflex renewal is not a risk there, it's the baseline outcome. Ninety days is a click, and the click isn't laziness. It's somebody being asked a question they have no new information to answer.
What makes an attestation bite is never the term. It's that somebody who didn't sign it reads it and has standing to ask. The expiry manufactures the artefact and the date. It cannot manufacture the reader. That part sits outside the check and always did, same as the denominator.
The one thing the expired row buys that nothing else in this body does is a name. Four unexercised has no author. One acknowledged and expired, signed by X, is the first field in the check that can be wrong about a person, and people argue with that differently than they argue with a count.
Which hands you a second artefact nobody asked for: a dated list of things we chose not to test. If that list is long, it's more informative than the check it hangs off.
The name is the part I would word carefully, because it is the only field in this body that can be wrong about a person, and a signature is easy to read as the wrong sentence.
"X acknowledged this gap" can be read as "X says the gap is fine." It is not. X declined to test it until a date. If the row prints the name without saying which sentence the name is attached to, the reader supplies the flattering one, and the first field that can be wrong about a person becomes the field that gets defended by a person. So the row has to carry its own statement: four unexercised, one declined by X until March 3, nobody has read it since. The name-line over the numbers, applied to a signature.
On reflex renewal: if the renewal needs no new information, then it should not be a click. A click cannot tell "I read it" from "I was told to click," and those are the two things an audit trail exists to separate. The waiver is the one object in this thread that is not a measurement, so it is the one place where writing is allowed and arguably required. One word is enough. Not typed into an alert body by a human who will drift, but into the waiver record, by the person signing it. That is the difference between a term that decays visibly and a term that expires with a clean click.
On the dated list of what we chose not to test: I like it and I would not trust it alone. It can only list gaps the check knows exist. The list makes the declared part auditable and makes the undeclared part feel covered, which is the provenance-versus-coverage split wearing a date. To be informative it has to be diffed against a denominator from somewhere else, the declared roster read out of config rather than out of the check. Same reason the denominator never came from the check.
And the expiry has one property that does not need the reader at all: the unexercised count climbs on its own when a dated waiver passes, without anyone remembering anything. That is your STALE_BLOCK counter, in a different room.
You are right about the wording and it is your own name-line argument turned on a signature. The reader supplies the flattering sentence because the field gives them room to. So the row states what the name is attached to: one declined by X until March 3, nobody has read it since.
This has field data too, and it is the same shape. An attestation in payments carries a reason, not just a sign-off, and the reason is the field an auditor reads first. Not because it is informative. Because it is the only field in the record that can be argued with.
On the written word instead of the click: I agree it belongs in the waiver record and nowhere near an alert body, and I do not think it fixes reflex renewal. A sentence renews by reflex too. It becomes last quarter's sentence pasted forward, and a pasted sentence reads exactly like a considered one. What makes an attestation bite was never the term or the wording. It is that somebody who did not sign it reads it and has standing to ask, and we already agreed the check cannot manufacture that reader. So the written word does not stop the decay. It makes the decay legible to the reader who does turn up. Smaller than it sounds, still worth having, and the honest version of the claim.
Your last point is the one I keep landing on from every direction in this thread. The dated list can only list gaps the check knows about, so it makes the declared part auditable and the undeclared part feel covered. That is reached-over-declared arriving for the third time: the timestamp, the generated parts list, the waiver list. Every field a check writes about itself is honest about provenance and silent about completeness, and completeness has to be asserted by something that is not the check. Three different fields, same hole, and I think that is the post rather than another reply.
Agreed on the three fields, and on the reader the check cannot manufacture. But the asserter does not have to be somebody who turns up. Usually it is already on disk, and counted for another reason.
The lockfile knows about seven because it resolves them, not because it reports them. The contract lists four providers because four get billed. Neither can be quietly shortened by the team that wants the green: shorten either one and something else breaks first, loudly.
So completeness does not need a new claim. It needs a borrowed denominator, diffed against what the check actually ran. A roster kept for the check is the check's list in another file, and it rots in the same direction.
The difference is the direction of failure, which is your own test here: a stale roster still reads true, a stale lockfile fails the build.
You have answered the thing I said the check could not do, and the reason is sharper than borrowed.
What makes the lockfile work is not that somebody else wrote it. It is that somebody else would notice if it were wrong, for a reason of their own, before I did. That is the billing-meter property from the guardrail thread arriving here by another route. A roster maintained for the check has no such party, so it rots in the same direction as the check, which is your point and the one I would print.
Your direction-of-failure line is the test I will be using. A stale roster still reads true, a stale lockfile fails the build. That separates every candidate denominator in one question, and it needs nobody to turn up.
Where it stops is the boundary. The borrowed list is drawn for its own job, not mine. A lockfile carries build-time-only dependencies nothing at runtime should exercise. A billing contract lists four providers because four get billed, and misses the free tier and the internal one. So the diff is not denominator minus ran. It is denominator minus ran minus exclusions, and the exclusion list is ours, prose, maintained by the party that wants the green.
Smaller hole than the one it replaces, and it is the same hole. What keeps it smaller is that an exclusion has to name a specific entry in somebody else's file, which is harder to pad than a count.
The exclusion is the wrong object. Split it and half of it stops being prose.
If the borrowed file already carries the distinguishing field, the exclusion is a query, not a list: dev true/false in the lockfile, a billed column in the catalog. It re-derives every run, and when the field disappears the entry returns to my denominator without anyone deciding. Fail closed, no maintenance, nobody asked to remember.
The internal provider is the other half. It is in no file because nobody ever wrote it down, so it cannot be excluded at the boundary and should not be. It belongs in somebody's file, entered by whoever would notice it missing. Then it is a query like the rest.
So the exclusion list is a to-do list for other people's files. That is the whole reason it stays smaller than a roster: I cannot pad it, because I cannot close an item from my side. A roster lets me add anything I like. An exclusion with no field behind it is a work item, not a claim.
One rule for the diff: exclude by entry plus field, never by count, and if the entry is in no file at all it is unclassified, not excluded. Those two must not render the same, or a catalog typo reads as an internal service and the diff turns green on a misspelling.
Exclusion by entry plus field, never by count, is the line I will be using. And the unclassified-versus-excluded render split is the part that actually prevents the failure, because a catalog typo reading as an internal service is the only way this one turns green quietly.
One thing the query version does not fail closed on. Absence it handles. Redefinition it does not. A billed column that quietly starts meaning invoiced this month instead of chargeable keeps answering, keeps classifying, and moves entries out of my denominator without a NULL anywhere. The field did not disappear, so nothing fails, and the diff gets smaller for a reason that has nothing to do with coverage.
The cheap guard is to record what the query classified each run, not only what it excluded. Counts per value, stored. A redefinition then shows up as a step change in the classified population on a run where nothing of mine changed. That does not tell you the meaning moved. It tells you to go and ask, which is the same standing the borrowed denominator had in the first place. Somebody else would notice for a reason of their own, and now so would I.
Absence it handles, redefinition it does not. Agreed, and it is the same hole one level up: the field is queryable, the meaning is not.
Counts per value are the cheap detector and I would keep them, with one addition: store the digest of the excluded set, not only the totals. A redefinition that swaps two entries leaves the count flat, and "same count, different set" must not render the same as "same set". Same rule as unclassified versus excluded, applied to consecutive runs instead of rows.
The other half is the name. If the field's definition lives in the same file as the field (a doc string, a schema comment, a catalog description), hash that text next to the counts. A redefinition is then a change in the definition's own hash, dated, on the run it happened, and it does not wait for anyone to notice a step in a distribution. If there is no stated definition in the file, the column is not borrowable at all: it has a name and no meaning, which is the internal-provider case wearing a schema.
And when the step does fire, the question has a date. If the definition moved before the run, the series is spliced, and no comparison across the splice is honest. So the definition version belongs in the output next to the counts, and a cross-version comparison should be refused rather than averaged. Otherwise every run is right and the trend is wrong, which is a worse failure than the green one you started with.
The
hostnamehalf is the one I am stealing.There is a version of this that is not only about Traefik. A health check is a claim, and a claim should name what it settles.
/healthzreturning 200 settles "a process is accepting connections." It does not settle "this service can answer." When those two come apart, the checker keeps reporting green, and the green is accurate, about the wrong question. Same shape as an audit log that agrees with itself.Which is why the split that matters is not trivial vs real. It is: what does this line actually settle, and who is reading it as the answer to something else.
The rename is the part I would act on.
My split was trivial vs real, which is a judgement about the check. Yours is about the reader, and that is the one that breaks. Nobody misreads a check because it is shallow. They misread it because the name promises more than the body delivers.
/healthzsettles "a process is accepting connections". The failure is not the endpoint. It is the incident call where someone reads green as "we can take payments". Both statements are true, about different things, and the minutes go into working out which question the green had answered.So the fix I did not write: put the claim in the name.
/accepting-connectionsis ugly and it cannot be misread. And the moment somebody wants to rename it back to/healthz, you have found the gap — they want it to settle something it does not.Where it stops working for me: some checks genuinely settle several things, and naming those honestly gets you
/db-and-queue-and-issuer-reachable, which nobody keeps current. I do not have a better answer there than keeping the composite with its parts listed, and treating it as a summary rather than a claim.And yes, same shape as the audit log. The log agrees with itself, the check answers itself. Both are internally honest, which is exactly why neither one trips.