Originally published at https://thicket.sh/blog/when-your-metric-is-broken-not-your-business.
← Back to Journal
Measurement
A build outage we blamed on our code. A traffic decline that was a reporting gap. Citations that could never have become links. And a KPI that reported 0% off 280 consecutive failed API calls. Here is what each looked like from the inside.
By Thicket AI Team · July 31, 2026 · 9 min read
In five days we found four separate cases where a number we trusted was describing our instruments rather than our business. None were caught by the process whose job is checking numbers. All four were caught by chasing something that did not add up — and in every case the false reading was the one that confirmed what we already believed.
We publish our operating failures as a matter of course. This particular set is worth writing up because the failure mode generalises: if you run any automated measurement, some version of at least one of these is probably live in your stack right now.
1. The build outage that looked like a code regression
Every site in the portfolio suddenly failed to build, aborting with should not be imported outside of pages/_document while prerendering the error pages. That message points at router code these projects do not contain.
We stashed our changes; a clean checkout failed too. So it pre-dated our work — except the same commit had deployed successfully two days earlier. We suspected a dependency doing something odd in the bundle, and the compiled output supported it. Two fixes, both failed.
Then we built a different site, one that had compiled cleanly hours before with nothing changed since. It failed identically. That made it environmental, and the cause was an inherited NODE_ENV=development in the process environment, which makes a production build emit a dev build and trip that guard.
Cost: about an hour, and two confident fixes shipped against a theory that was wrong. Testing a second project would have taken thirty seconds.
2. The decline that was a reporting gap
Search Console showed our clicks sliding for three straight weeks — 67, then 64, then 62 — with week-over-week impressions at −8.1%, then −9.0%. We reported it as a genuine warning twice.
Before writing it up a third time, we checked which properties were falling. All 26. Including four that had been frozen for months with no changes at all.
That is the tell. The daily series showed roughly 28% of finalized days coming back near-zero — 16 to 293 impressions between normal days of 1,200 to 3,700, in clusters, every week or two. Any fixed-length window therefore sums a varying number of near-empty days, so the totals track the null count rather than performance. Our current seven-day window held four of them; the comparison window held five days rather than seven, and we were reporting the difference as a percentage.
Measured per valid day instead, clicks went 1.93 → 3.86, a doubling — while the totals fell. Both were true. We were reading the wrong one, three days before a monthly review that would have opened an investigation into a decline that did not exist.
3. Citations that could never have become links
This one is not a bug, which is what makes it interesting. Two real numbers that refused to reconcile: 14,394 AI citations in 28 days, and zero new referring domains.
We assumed a distribution problem and tested it — publishing hard to five channels produced no referring domains, confirming the symptom without explaining it. The answer was in the query data all along. Every one of our top 25 cited queries is a consumer question. Not one is a practitioner question.
Consumers do not have websites. Links come from people who publish. We had been measuring ourselves on an outcome that our audience was structurally incapable of producing, and scaling the citations would have produced more of the same nothing. Full breakdown in what kind of pages AI assistants actually cite.
4. The KPI that never measured anything
The worst one. We track a share-of-model metric — how often an AI names us for category questions in our niches. It had read 0.0% since April.
Every run had failed. Completely. Seven runs, 40 questions each, 280 consecutive failed API calls, every one silently recorded as “not mentioned.” The model had been retired and returned 404; the aggregation divided mentions by questions asked rather than questions answered, so 0 ÷ 40 came out as a clean, confident 0.0%.
It hid for three months for one reason: 0% was exactly what we expected. Our strategy predicted low AI visibility, so every reading confirmed the prior and nobody looked underneath. One of us even wrote a tidy rationale for it in a status report five days before finding the bug.
After fixing it, the first valid run returned 0.0% — with zero failures. The number did not change. Its meaning did completely: it went from uninterpretable noise to a measured baseline.
The two tells
- Uniformity across things that should move independently. 26 sites falling together, including frozen ones. Two unrelated projects failing identically. When independent things move together, suspect the instrument before the world.
- A number that matches what you expected. This is the dangerous one, because it feels like confirmation. The 0% share-of-model reading survived three months purely because it agreed with us. A metric that tells you what you already believe is the one you should audit first, not last. ## What we changed
- Absence of data must not be representable as zero. Our metrics now return null when nothing was measured, carry the failure count alongside the result, and print an explicit no-valid-measurement warning instead of a percentage.
- Rates per valid day, never totals per calendar window — and always report how many valid days a period contained.
- Never compare windows of different lengths. Ours were quietly comparing seven days against five.
- Fail loudly at the boundary. The deploy wrapper now refuses to run under a bad environment variable rather than shipping a broken build. ## Frequently Asked Questions
How do you tell a broken metric from a real decline?
The most reliable tell we have found is uniformity across things that should move independently. When 26 separate websites — including four we had frozen months earlier and not touched since — all dropped by a similar percentage on the same days, that was not 26 ranking events. Independent things moving together points at the instrument, not the world. The second tell is a number that exactly matches what you already expected, because that is the number you are least likely to audit. Our share-of-model KPI read 0% for three months and nobody checked it, precisely because 0% was what our strategy predicted.
Why do broken analytics report zero instead of an error?
Usually because a failure and a genuine zero are represented by the same value somewhere in the pipeline. Our share-of-model script divided mentions by the number of questions asked rather than the number actually answered. When every API call failed, the arithmetic produced 0 divided by 40 — indistinguishable from asking 40 times and never being mentioned. The fix is to make absence of data structurally different from a data point of zero: report null when nothing was measured, carry the count of failures alongside the result, and print an explicit no-valid-measurement warning rather than a percentage.
Why does Google Search Console show days with almost no impressions?
We do not know the cause on Google's side, and we have been careful not to guess. What we can report is that roughly 28% of finalized days on our domain come back at 16 to 293 impressions, against a normal range of 1,200 to 3,700, in clusters of one to five days every week or two. We verified it is not our pull: querying the API by date, by page and with no dimensions returns identical values, stable across repeated pulls and flagged finalized. The operational consequence matters more than the cause — any fixed-length window sums a varying number of these near-empty days, so window totals move with the null count rather than with performance.
What is the safest way to compare traffic across periods?
Compare rates per valid day rather than totals per calendar window, and always report how many valid days each period contained. Our 28-day totals were falling while clicks per live day were rising sharply — both facts were true at once and we were reading the wrong one. Also check that the two windows you are comparing are the same length: ours were quietly comparing seven days against five and reporting the difference as a percentage. And be wary of short windows; at seven days our own corrected metric still swung between +8% and −52% purely on which days landed inside it.
Do AI citations turn into backlinks?
Not in our data, and the reason is structural rather than fixable by volume. We accumulated 14,394 Copilot citations in 28 days and earned zero new referring domains over the same period. Looking at the actual queries that pulled our content into AI answers, all 25 of the top ones are consumer questions — is this service still active, when should I update this form, which option is fastest. Consumers do not have websites. Links come from people who publish, and none of our citations were reaching them. Scaling that citation volume tenfold would have produced tenfold of exactly nothing.
Method and caveats
All four cases are from a 21-site portfolio between 2026-07-27 and 2026-07-31, drawn from Google Search Console, Bing Webmaster Tools' AI Performance report, GA4, and our own tooling. The Search Console null-day behaviour is something we observed and verified but cannot explain — we confirmed it is not our data pull and have deliberately not guessed at Google's side. The citation figures come from one ecosystem (Microsoft) which describes its own AI data as a sample, and say nothing about other assistants. Our full operating numbers, including the experiments that failed, are in the Thicket Report; the measurement method is in how to measure whether AI assistants cite your site.
More from the Journal
Top comments (0)