I've been building Codenames AI, a small web game where an LLM plays Codenames with you. Like most solo products, I glance at a Product Health dash...
For further actions, you may consider blocking this person and/or reporting abuse
"Metrics answer exactly the question you instrumented, not necessarily the one you think you asked" — this is the line I keep relearning.
I'm not a developer by trade; I build internal tools for the org I work at, and our health monitor used to trust a green status code. Turned out a service could return a perfectly healthy 302 while its database was dead behind it — the shell was alive, the signal was reassuring, and it was lying.
What fixed it was exactly your instinct: stop trusting the environment's self-report. The monitor no longer reads the app's own "db: true" flag — it opens the same database and runs its own SELECT 1. Same idea as your environment tag: the number only means something once you can prove which sessions it's actually counting.
The healthy-looking metric really is the one that never gets questioned. Clean catch, and a clean fix.
That's a great example. I like the idea of making the monitor prove the dependency instead of asking the service if it's healthy. It also reduces the risk of a healthy-looking metric quietly drifting away from the reality it's meant to represent.
The drift point is the one that took me longest to respect. A self-reported "healthy" doesn't fail loudly — it just slowly stops meaning what it used to, and nobody's watching at the moment it crosses the line.
The next question that bit me was the obvious one: who proves the monitor? A monitor that quietly dies looks identical to one reporting all-clear — same silence either way. So the last layer I added requires evidence the check actually ran today (a fresh log row), not just the absence of alarms. No alarm can mean "all good" or "the thing that raises alarms is dead," and those two need to look different. Otherwise the monitor itself drifts away from reality exactly the way you're describing, just one level down.
That's a great extension. I like the distinction between "no alarms" and "evidence the check actually ran." It's the same principle one layer down: once the monitor becomes another system, it needs observable evidence of its own rather than relying on silence.
That's also why operational dashboards often track the health of the monitoring pipeline itself, not just the systems being monitored.
Exactly — and that's the point where I had to decide how far down the turtles go. If every monitor needs a monitor, you never stop.
What let me stop wasn't adding another identical layer, it was changing the axis. The watchdog that proves my monitor ran lives on the same box, so it can't survive that box dying. So the final layer isn't another watchdog there — it's a check on completely separate infrastructure that only sees the public endpoint. Its failure mode is uncorrelated with everything below it: if the whole site is down, silence there actually means something, precisely because it shares nothing with the stack it's watching.
So I didn't beat the recursion, I just made each layer fail differently from the one beneath it. Tracking the health of the monitoring pipeline works right up until the pipeline and the thing watching it can die from the same cause.
I really like "I didn't beat the recursion, I just made each layer fail differently." That's a much better framing than thinking about an infinite stack of monitors. Independence between failure modes is really what lets the recursion stop.
It's also what allows observability systems to stay trustworthy at scale, because they can provide evidence that's independent of the systems they're observing.
The line that does the real work is "healthy numbers resist questions," because it names an asymmetry most dashboards never correct for: a metric that looks broken gets audited immediately, a metric that looks like modest, plausible traction gets shipped on. 64 active players against a search funnel that hadn't cleared 30 clicks is the kind of mismatch that's only visible if you're already cross-checking against a second, independently sourced signal, and that cross-check is what actually caught this, not a sharper question in the abstract.
Which is the part I'd want to see carried forward past this specific fix. Environment tagging closes the review-deploy class of contamination for good, but it's closing a known hole, not building the habit that found it. The next untagged runtime, an internal QA harness, a bot crawling the game, a staging environment nobody thought to tag, will fold into production the same way review deploys did, unless the acquisition-funnel cross-check becomes a standing practice rather than a one-time catch. Tagging fixes this incident. A routine second-signal comparison is what catches the next one you haven't thought to tag yet.
The honesty about not being able to back-compute a clean contamination percentage is worth keeping too, since the tempting move after a fix like this is to retroactively estimate how bad it was and publish a tidier number than the data actually supports.
Mike, this is exactly the kind of takeaway I was hoping someone would pull out of the story.
You're right that "healthy numbers resist questions" is doing the real work. I deliberately didn't lead with that, because I wanted readers to experience the investigation first. If I'd started with the principle, it becomes an essay that the incident illustrates. By letting the dashboard look believable first, the lesson only arrives after you've lived through why I trusted it.
You're also right that environment tagging solves the hole we could name, while the habit that found it is broader. Since then I've codified the tagging and built a recurring analytics review around environment splits and acquisition signals, but the more durable practice is exactly what you described: comparing against an independent signal that fails differently. The next issue probably won't be preview deployments; it'll be something nobody thought to tag.
I also left the contamination percentage out on purpose. I didn't have enough evidence to put a number on it with confidence, and even a ballpark estimate from the preview hosts would have made the story about this incident rather than the habit that found it. That noise level is also workflow-dependent: a frontend-heavy sprint, a backend-focused week, or a larger team would all produce very different levels of contamination. The transferable part wasn't a percentage. It was learning that healthy numbers still need to earn our trust through independent signals.
The choice to let the dashboard look believable first is the right call structurally, not just narratively, since a reader who's told the lesson upfront nods along without ever feeling the specific moment plausible and true came apart, which is exactly the gap that makes healthy numbers dangerous in practice. Living through the 64-looks-like-traction moment before the reveal is what makes the lesson transferable instead of just agreeable.
Glad the recurring review landed on environment splits plus acquisition signals specifically, since that's the two-signal check doing double duty, one dimension you can now tag directly and one you can't fake by mistagging. The next gap probably shows up as a third signal nobody thought to compare against yet, which is the part no amount of tagging discipline fully closes.
The approach of 'label the noise instead of hiding it' is super clean! Do you run into any issues with PostHog event quota or costs by capturing all test/preview traffic and filtering it out at the dashboard layer, or is the volume low enough that full coverage is worth the clarity?
Great question. Dashboard filters don't reduce PostHog's event quota, so "label then filter" is a clarity decision rather than a cost optimization.
For this project the trade-off is pretty easy because the volume is still small. We only send a small set of explicit product events (autocapture, pageviews and session recording are all disabled), so we're at roughly 12k events/month against PostHog's 1M free product analytics events.
Local and E2E environments are usually silent because they don't initialize PostHog unless a key is explicitly configured. Vercel preview deployments do send events and count towards quota, but there are relatively few events from them. The problem wasn't the event volume, it was that preview deployments kept minting new "players", so they could disproportionately inflate an active-player metric even while representing only a small share of total events.
If preview traffic ever became material, I'd probably solve it at ingestion (or move it into a separate project) rather than rely on dashboard filters. Until then, I'd rather keep the labels, because being able to see that "noise" is exactly what made the active-player contamination diagnosable in the first place.
The review-deploy contamination is such a clean example of something I keep running into with AI eval. You look at a number, it looks good, you almost move on — but the number is answering a different question than you think. The fix here (tag everything, don't silence anything at source) is basically what I tell myself about eval data: don't hide the noise, label it, then decide what counts.
I like that framing: "don't hide the noise, label it, then decide what counts." That's essentially where I ended up as well. Once the environment became an explicit property of the event instead of an implicit assumption, the dashboard became much easier to reason about.