What a refractometer taught me about observability
In July, I wrote about a farmer walking a vineyard row at dawn, crushing a grape onto a refract...
For further actions, you may consider blocking this person and/or reporting abuse
Absolutely. A flat error-rate line without request volume can be deeply misleading. Zero errors because 100,000 requests succeeded and zero errors because nobody made a request are very different observations.
I think that's another version of the same underlying problem: the metric needs enough context to preserve what the observation actually meant. Otherwise, “0% errors” quietly gets promoted into “healthy,” even though the system may not have demonstrated much of anything.
On a follower count, a flat day can hide one follow and one unfollow, so our daily snapshot keeps the list of follower IDs next to the total and gets follows and unfollows by diffing it against the previous snapshot's list. That only sees what changed from one snapshot to the next, though: someone who follows and leaves in between never appears in either list.
"Your metric is not your state" is the sentence I wish I had framed on the wall about two years ago. I lost weeks to a version of exactly this, and the domain made it expensive rather than merely annoying.
The setup: a risk guard in a trading system. Its metric was "positions currently open, and the margin they consume". Its state was "the account, right now, as the broker sees it". The metric came from a snapshot that a different code path refreshed. When that path quietly stopped refreshing - no exception, no error log - the metric froze at a plausible-looking value. The guard kept evaluating its condition, kept passing, and the state it was supposed to protect drifted away from it.
What made it invisible is what you describe: the metric was never wrong in a way that looked wrong. A frozen number is not obviously frozen. Zero is not obviously "no data". Everything downstream was internally consistent.
Three things I do now, all cheap:
Separate "healthy" from "no data" explicitly. If the guard cannot read its input, that is an incident, not a zero. A missing measurement and a measurement of zero must never collapse into the same value, because only one of them means "fine".
Assert on the state, log the metric. Tests that assert a freshly computed metric prove the formula, not the system. The test that catches this asserts the object production actually reads - and a single log line per cycle with that value turns a silent freeze into something you can see.
Ask what the metric's refresh path is, and whether anything checks it. Most of these bugs are not "wrong computation", they are "stale input". A freshness assertion (timestamp + threshold) is one line and catches the whole family.
The generic version, for anyone who has not had the pleasure yet: metrics describe the system you instrumented; state is what the system is. Every gap between those two is a place where a monitor happily reports green while the thing it watches is already gone.
I wrote up the trading variant of this (with the persistence and freshness fixes) here, if it is useful: xuks124.github.io/vigildesk/blog/s...
Good post - the distinction deserves to be stated this bluntly more often.
“The metric was never wrong in a way that looked wrong” is exactly the dangerous case. A spectacularly broken number tends to announce itself; a plausible stale value can keep authorizing decisions long after its relationship to reality has disappeared.
I especially like the freshness assertion for that reason. It adds another piece of evidence about whether the measurement is still entitled to mean what we're treating it as meaning. And your zero/no-data distinction gets at the same problem from another direction: absence of an observation cannot safely collapse into an observed state.
Thanks for the trading example. That's a much higher-consequence version of the same failure than my misbehaving glass of blackberry wine.
The refractometer analogy is a good one for cost dashboards specifically, a metric that looks healthy can still be measuring the wrong proxy for what's actually driving the bill
What's the metric you've seen most commonly mistaken for the real state it's supposed to represent?
Model confidence probably bothers me most right now.
confidence: 0.94has an almost irresistible tendency to become “94% chance this is correct,” even when that's not what the number establishes at all.Error rate is another good one. A dashboard showing zero errors looks wonderfully healthy until you discover there were zero requests. That's the same refractometer problem in miniature: the measurement may be perfectly accurate while the interpretation is completely wrong.
I think that's the pattern I'm becoming more interested in. The dangerous metric isn't necessarily a bad measurement. It's a good measurement that has quietly been promoted into evidence for a state it doesn't actually establish.
Cost dashboards fit that beautifully too, especially once “this span incurred $X” gets interpreted as “this span caused $X.”
"Incurred $X" quietly becoming "caused $X" is exactly the gap that breaks cost attribution on shared infrastructure, a span can be billed for its share of an always-on resource it barely touched
How do you handle that in practice, do you flag correlation-based cost allocation differently from direct causation, or does it all get reported the same way once it hits a dashboard?
I'd flag them differently, absolutely. I think "incurred," "allocated," and "caused" are three different claims, and a dashboard that collapses them into one dollar figure is quietly asserting more than the evidence supports.
For a directly metered API call, I can usually establish a fairly strong relationship between the operation and its charge. For shared, always-on infrastructure, I might allocate cost by CPU time, requests, tokens, or some other weighting rule, but that tells me how I divided the bill, not necessarily what caused it. Removing that workload might save nothing until the infrastructure can actually be downsized.
In practice, I'd want the reported amount to carry its attribution method, the underlying observations, and the assumptions used to calculate it. I'd also distinguish allocated cost from estimated avoidable cost, because they're useful for different decisions.
That's really the same distinction I was trying to make with the refractometer: preserve what was measured, preserve how it was interpreted, and don't promote the interpretation into a fact the instrument never established.
Curious whether you've seen dashboards that make this distinction explicit, or whether most simply report everything as "cost by workload."
Calibration drift has a neighbor that hides inside a single record. Volatile fields can borrow credibility from durable fields sitting next to them in the same JSON object. In an agent-directory API I scanned,
last_seenandevent_countcame from an append-only signed ledger.is_healthyanduptime_24h_pctcame from a 30-minute heartbeat window held in process memory, which a restart zeroes. Nothing in the response marks which is which.That puts a condition on "keep the raw reading and re-derive later." An unpersisted window has no raw reading to keep. A window that was never written cannot be recomputed, so provenance only holds if the observation outlives the process that took it.
The two halves also never contradict each other. A stale ledger does not pull the health number down, and a silent heartbeat does not flag a key that is still signing. Two fields that cannot disagree cannot catch each other being wrong.
Numbers from that scan: across 60 keys,
uptime_24h_pctcan take 97 values (96 fifteen-minute buckets), and only two ever appeared. 100.0 on 15 keys, 0.0 on 45. Nothing in between, because anything that beats at all beats on a tight loop and fills every bucket. Three of the 15 keys showing 100.0 had last signed an event 99, 121 and 126 days earlier.Cheap check before any provenance work: count how many distinct values a supposedly continuous metric has ever taken. That is a distribution test, separate from watching a single line go flat.
Is there a workable way to make a field declare whether its evidence survives a restart?
Yes, I think there is, and you've identified a prerequisite I mostly skipped over in the article: before I can preserve provenance for an observation, the observation has to survive long enough to have provenance.
I'd be tempted to make evidence durability explicit metadata rather than something readers have to infer from the field name. Something along the lines of
evidence_scope: durable | process | window | derived, although I'd want to think harder about the vocabulary. At minimum, I'd want to know the observation source, retention boundary, window, and whether the underlying evidence can be reconstructed after restart.Your example also makes me uncomfortable with putting
last_seen,event_count,is_healthy, anduptime_24h_pctside by side without qualification. JSON structure gives them visual equivalence even though their evidentiary properties are radically different. Two numbers sitting next to each other can look equally durable, equally historical, and equally recomputable when none of those things are actually true.And I really like your “can they disagree?” test. Independent-looking signals aren't providing much cross-checking value if their definitions make disagreement impossible or if they observe unrelated windows. Three keys being reported at 100% uptime while having produced no signed event for 99, 121, and 126 days is exactly the sort of result that should make us ask what “uptime” is actually entitled to mean.
The distinct-values check is wonderfully cheap too. Before building elaborate provenance machinery, ask whether your supposedly continuous metric has enough information content to discriminate between interesting states in the first place. If 60 entities collapse into only
0.0and100.0, the problem may begin considerably earlier than the dashboard.So I think my answer is yes: evidence lifetime belongs in the field's contract. A consumer should be able to distinguish “derived from durable observations I can revisit” from “derived from process-local state that disappears on restart.” Otherwise, the API is exporting values while hiding an important part of what those values are evidence of.
You've given me another dimension I didn't have in my little raw/corrected table. I'll have to think about this one.
the raw/corrected split has another payoff: you can ask whether last week's "done" would still be done under today's evidence rule. if the formula changes but the dashboard overwrites the old conclusion, you lose the ability to find decisions made under the bad calibration. keeping the rule version beside the reading makes that review possible without pretending you knew it at the time.
Yes. I think you're exactly right that the correction rule itself belongs in the provenance.
Keeping raw + corrected lets me recompute the interpretation later, but if I don't preserve which rule produced the original corrected value, I've lost something important: what the system believed at that time.
That gives you two different historical questions:
Overwriting the corrected value collapses those into one and quietly rewrites history. Versioning the interpretation rule lets both remain answerable.
I had this with entity resolution. A similarity score I'd tuned carefully kept returning confident values after we switched data vendors, and the merge quality had been quietly degrading for weeks before a downstream analyst complained about the duplicates. Looking back at the logs, every metric had looked fine the whole time. The score wasn't wrong; it had just stopped being a proxy for what I was actually trying to guarantee.
This is a fantastic example of the distinction I was trying to get at. The similarity score didn't necessarily become a bad measurement. The relationship between that measurement and the thing you actually cared about, merge quality, had changed.
And the vendor change is especially interesting because it gives you a boundary event. In hindsight you'd want to ask whether the old calibration remained valid on the new input population rather than assuming continuity because the metric itself kept producing plausible-looking numbers.
“The score wasn't wrong; it had just stopped being a proxy for what I was actually trying to guarantee” may be the whole problem in one sentence. Thanks for sharing this.
hold up. green settlement tiles are not a signed hop tip.
1 cut: when chargeback week opens, can anyone GET the queryable tip after the vendor UI flips, or only another dashboard seal?
receipts > seals. marker0929h2128-dt
I think I follow the distinction you're making: a green settlement status is an interpreted state, while a durable, queryable receipt preserves the evidence behind that state. If so, that maps closely to what I'm arguing here: keeping the observation separate from the conclusion.
I'm not familiar with “signed hop tip” or “dashboard seal,” though. What do those mean in the system you're describing?
The refractometer example is a great pick because the reading isn't wrong, it's measuring the wrong thing once alcohol bends the light. The software parallel is exactly the trap, HTTP 200 means a response came back, not that it was correct, same way 15% CPU can be a deadlock. What's your go-to for catching the case where every dashboard is green and the system is still wedged?
My go-to is some form of independent end-to-end observation that exercises the property I actually care about, rather than another internal health signal.
If it's a web application, that might be a synthetic transaction that logs in, performs a real workflow, and verifies the resulting state. If it's a data pipeline, I might inject something known at one end and verify that the expected artifact emerges at the other. The important part is that I'm not asking the same system that produced all the green lights to give me one more green light.
Even that has an observation boundary, though. A synthetic transaction can prove that the transaction succeeded under those conditions. It doesn't prove the entire system is healthy. That's why I like pairing it with things such as throughput, queue depth, age-of-work, and actual outcome signals where they're available.
I think the general rule I'm converging on is: when every dashboard is green, ask what externally observable consequence should be happening if the system really is healthy, then measure that independently.
And if I can't name that consequence, that's probably telling me something uncomfortable about what “healthy” means in the first place.
The distinction between a measurement and the underlying reality is a hard lesson to learn in distributed systems. I've seen teams obsess over p99 latency metrics while the actual user experience was degrading due to client-side execution issues that the backend telemetry simply couldn't capture. It's easy to fall into the trap of optimizing for the dashboard rather than the actual state of the system. One thing I've learned is that a metric is just a lagging indicator of a state that has already passed. If you aren't correlating your metrics with distributed traces or actual event logs, you're basically just looking at a high-resolution photo of a ghost. You see the shape of the problem, but you have no idea what actually caused it.
That's a great example of the observation boundary being the real problem. The backend latency metric can be perfectly accurate about what it observes while saying nothing about the client-side state the user actually experiences. I think that's what makes these failures tricky: the number doesn't necessarily become wrong. We quietly start asking it to support a claim it was never capable of establishing.
The refractometer shows it well. The instrument works, the system changed, and the proxy stopped being valid.
I had the same failure in web analytics. My dashboard counted 336 "human" views in one day. Two cloud IPs were probing for .env files and a few AI crawlers had slipped past the bot filter. After I removed those, about 47 were real reads. The counter didn't break. The assumption behind "human" did.
What caught it was a second signal: a JS beacon that only real browsers fire. How do you notice a proxy has stopped being valid before the correction formula disagrees?
I think your second signal is actually a big part of the answer: independent measurements that fail differently. If the counter and JS beacon normally maintain some relationship, divergence becomes evidence that one of your assumptions deserves inspection even before you know which one failed.
The other thing I'd want is an explicit validity condition for the proxy itself. Not just “336 human views,” but effectively “336 views classified as human under filter/version X, with these known observation limits.” Then changes in the environment, classifier, collection path, or expected relationship between independent signals can invalidate the interpretation even when the number still looks perfectly reasonable.
Which may be the nastier lesson here: sometimes you can't detect proxy failure from the proxy itself.