Vicente Reyes wrote up a deploy debugging story this week. Ten small breakages stacked in series, each one hiding the next. Good piece. Go read it....
For further actions, you may consider blocking this person and/or reporting abuse
The composite is the honest part, so I would not try to name it precisely. I would date it.
/healthz-or-whatever-it-is-called settles one thing: "this summary was accurate as of the check that last ran." That sentence is true no matter how many parts you fold in, and it can be kept true by the check itself, not by the person maintaining the repo. The parts list is what degrades, so the parts list has to live where the reader is at 3am, which is the alert body, not the runbook nobody opens.
The part of your reply I keep turning over is the reader. You are right that the rename is a judgement about the reader, and that is why it is not fully in your hands: the person who reads green at 03:14 has never seen the check, the runbook, or this post. They read the name as the answer to the question they already had. Which means the name is not really a name. It is a claim with a population, and the population is whatever the reader assumes it covers. /healthz says "one process is listening" and gets read as "we can take payments". Nobody miscounted. The sentence swapped the population.
That is why I would watch for the moment of renaming-back you named, and treat it as the useful signal it is: not a preference about filenames, but somebody wanting the check to cover a claim it does not make. The question I would put to them is not "why the old name" but "what do you want green to mean when the phone rings". If they can answer that in one sentence, you have the real check and the name follows. If they cannot, no filename saves the incident.
"A claim with a population, and the population is whatever the reader assumes it covers" is the sentence. Nobody miscounted. The sentence swapped the population. I had this filed as a naming problem and it is not one, because a name cannot carry a population at all.
The date works for a reason worth naming, because it is the test for everything else. "Accurate as of the check that last ran" stays true with nobody maintaining it. The check writes the timestamp. There is no edit anyone can forget.
Run that test on the parts list and it fails, including in the alert body. A written parts list drifts the first time somebody adds a dependency to the check and not to the string, and nothing breaks when they do. Same failure class as the check itself. Moving it into the alert body does not fix that, it moves it somewhere more people see it being wrong.
The version that passes your test is a parts list generated from whatever the check iterates, so there is one list and both the check and the message read it. If it has to be typed by hand I would rather it were not there, because a stale parts list is worse than none: it reads as the population and it is out of date.
Where I think the argument stops is the 03:14 reader you put at the centre of it. The parts list in the alert body is not read at 03:14 either. They read the colour. The timestamp survives that and the parts list does not, and not because of where it lives. A stale date is visible without parsing anything. A missing dependency is not.
The parts list is where I was wrong, and you found the reason: I moved it, I did not generate it.
Anything a human types into the message drifts, and a stale population is worse than no population, because it reads as the answer. One object, iterated by the check and rendered into the message, is the only version of that list I would defend. If it has to be typed by hand, leave it out.
What I would keep is a smaller claim than the one you are testing. Not "the list is accurate" but "the list is the thing that ran". A generated list cannot be more complete than the check, which is fine. It cannot be wrong about what the check did, which is the point. Same shape as your timestamp: the property comes from the fact that nobody writes it.
And on the 03:14 reader, you are right and I was arguing the wrong defence. They read the colour. No list in any body reaches them. That is why the rename is load-bearing and the parts list is not borrowed by them at all. The reader who gets the list is the one at 09:00 opening the incident, and their question is not "what is this" but "what did it cover". Different reader, different artefact. My mistake was dressing the 09:00 list up as the 03:14 fix.
Your smaller claim is better than the one I was testing, and I'll take it: the list can't be wrong about what the check did, and that's a provenance claim, not a coverage one. Provenance is the only thing a check's own body can honestly carry.
Where it still lets someone through: a generated list is unfalsifiable in the direction that hurts. The check iterates three of seven dependencies, the list prints three, every word of it is true, and nothing in the message knows seven was the number. Correct about what ran, silent about what didn't. So the generated list buys you "this is real" and not "this is all" — and the second one is what the 03:14 reader would want, if they were reading.
Which puts a count from somewhere else back into the message: reached against intended, where intended comes from the service registry or the dependency manifest rather than from the check. Same shape as your timestamp, one layer out. The denominator has to be written by something that isn't the check.
And yes on 03:14. That's the argument for the rename being the whole fix rather than part of it. If colour and name are the only two fields that reach a person mid-incident, every other field in that body is written for the post-mortem on Tuesday morning. Worth saying out loud, because most health check bodies are written as though someone reads them while paged.
You are right that the denominator cannot come from the check. I think the fraction you build out of it fails the same test one layer further out.
3 of 7 declared, rendered as a number, is a ratio, and a ratio invites a threshold. Whoever decides that four unexercised is amber and three is red has made a policy call inside an artefact that is supposed to be a measurement. Same coat as an invented cost line: a number the reader treats as measured because it printed.
What I would keep from your move is the direction of falsifiability. Your numerator is checkable against the check's own body. If the message says three and the code iterated four, someone can catch it. Your denominator is checkable against nothing except the manifest it came from, and the manifest describes what was declared, not what was required. So the ratio is reached over declared, and the sentence a reader hears is reached over required. A wrong numerator is a bug. A wrong denominator is a belief.
Which is why I would not render the fraction at all. Render the gap as its own field, and make it non-aggregating: "four declared dependencies not exercised by this run". That is either zero or it is not, it can never turn green, and it needs no threshold to be read. A count of absences survives drift better than a proportion of them, because nothing rounds a gap up.
The ratio goes. You're right about why, and it is the cleanest version of the argument so far: a threshold is a policy call wearing a measurement's clothes, and once the number prints, somebody tunes it. I was going to put a fraction in a field and then act surprised when it grew an amber band.
"A wrong numerator is a bug. A wrong denominator is a belief." That is the line I will be repeating.
What the gap field does not buy, though, is escape from the denominator. "Four declared dependencies not exercised by this run" is reached-over-declared with the division left out. The belief is still in there. It just cannot be rounded. So the win is smaller than survives-drift-better. The win is that nobody can tune it, and tuning was the failure mode.
Where I think it breaks: that field will be non-zero forever on a real service. Some dependencies are declared and genuinely do not belong in this check's claim. A field that never reaches zero gets learned around in about three weeks, same as an amber band set too wide. Permanently non-zero and ignored is the same outcome as a threshold, by a different route.
The only fix I can see puts policy straight back in. The absence has to be acknowledgeable per dependency, with a name on it. Not "four unexercised" but "four unexercised, three signed off by someone, one nobody has looked at". Policy does not leave the artefact. It stops being a threshold and becomes attributable. A band hides whose judgement it was. A waiver has an author.
Then the part I have no answer for: nobody keeps those waivers current. It is your parts list again, one layer out, with the same property that nothing breaks when it goes stale. The only difference I would claim is that a stale waiver has a name attached, so there is someone to ask. That is weaker than generation. It is the best I have where the fact in question is a human decision and cannot be derived from anything the code iterates.
The
hostnamehalf is the one I am stealing.There is a version of this that is not only about Traefik. A health check is a claim, and a claim should name what it settles.
/healthzreturning 200 settles "a process is accepting connections." It does not settle "this service can answer." When those two come apart, the checker keeps reporting green, and the green is accurate, about the wrong question. Same shape as an audit log that agrees with itself.Which is why the split that matters is not trivial vs real. It is: what does this line actually settle, and who is reading it as the answer to something else.
The rename is the part I would act on.
My split was trivial vs real, which is a judgement about the check. Yours is about the reader, and that is the one that breaks. Nobody misreads a check because it is shallow. They misread it because the name promises more than the body delivers.
/healthzsettles "a process is accepting connections". The failure is not the endpoint. It is the incident call where someone reads green as "we can take payments". Both statements are true, about different things, and the minutes go into working out which question the green had answered.So the fix I did not write: put the claim in the name.
/accepting-connectionsis ugly and it cannot be misread. And the moment somebody wants to rename it back to/healthz, you have found the gap — they want it to settle something it does not.Where it stops working for me: some checks genuinely settle several things, and naming those honestly gets you
/db-and-queue-and-issuer-reachable, which nobody keeps current. I do not have a better answer there than keeping the composite with its parts listed, and treating it as a summary rather than a claim.And yes, same shape as the audit log. The log agrees with itself, the check answers itself. Both are internally honest, which is exactly why neither one trips.