A few weeks ago I killed an indicator of mine in public.
I had been trying to work out how much of my audience was automated. One signal was whether an account had uploaded its own avatar. It fired on 100% of the accounts I was confident were people and 97% of the ones I suspected were not. That isn't a lenient signal. It isn't separating anything — it tracks something both groups share, and I had been counting its votes for weeks.
I wrote that up. Named the defect, retired the indicator, moved on feeling like I'd learned something.
Three days later I shipped another one.
The same hole, in a different shape
I needed to check whether a comment on one of my posts was actually visible to readers — I'd found one the API returned and the comment count included, but that moderation had removed. So I wrote a check:
// v1 — passes for anyone with a second comment on the page. Zero separation.
visible: html.includes(comment.user.username)
// v2 — the only witness with jurisdiction over one comment.
visible: html.includes(comment.id_code)
Two comments from the same account, one removed and one live, both came back visible under v1. I found it by accident, and only because I happened to compare against something else.
Someone in a thread put the problem in a sentence I couldn't argue with: if the fix is "I noticed this one," the next indicator ships with the same blind spot in a different shape. Which is, word for word, what I had already written about the previous defect.
Their prescription was structural. A labeled control set shouldn't be a diagnostic you run when something feels off. It should be a permanent seed every indicator has to clear a margin on before it's allowed to vote — not just beat chance on the live population, which is exactly the condition that let the avatar signal pass silently.
Building it
Twenty-eight accounts. Fourteen labeled human, fourteen automated, and every label carries a provenance string saying how it was established — seven humans confirmed through long back-and-forth exchanges, the rest by a weaker proxy. That doesn't make a label true. It makes an invented one visible, which is a different and more achievable thing.
Then a margin: 30 points of separation before an indicator votes.
indicator human automated margin
github/twitter 64% 7% 57pt admitted
avatar uploaded 100% 100% 0pt rejected
location present 57% 0% 57pt admitted
The dead indicator died on contact. That's the class the gate was built for, and it works.
Testing the threshold the way I tested the indicators
Thirty was a number I picked because it sounded right. Having just written a whole piece about constants chosen by feel, I could hardly leave it there — and the same person named the obvious move: pressure-test the threshold exactly the way you pressure-tested the indicators.
Shuffle the labels. Recompute every margin against the scrambled pairing. Twenty thousand times, with a fixed random seed so the result is reproducible rather than remembered.
observed null p95 null p99 p
github/twitter 57pt 29pt 43pt .0034
avatar uploaded 0pt 0pt 0pt 1.000
location present 57pt 29pt 43pt .0011
bio present 57pt 29pt 43pt .0055
website present 71pt 43pt 43pt .0002
Thirty sits at the 96.4th percentile of that null. The 95th is 29 points, the 99th is 43. My number was approximately right by luck, and it's measured now instead.
Except moving it to 43 changes nothing. Every observed margin is either 0 or between 57 and 71. Nothing lands in the gap. The constant is now calibrated and has still never decided anything — it only starts mattering the first time an indicator falls between those values, and until then "calibrated" is a property I can claim without having tested it.
The part I didn't see coming
Location present: p = .0011. Second strongest of the five.
Location is the indicator I threw out weeks ago. An empty location field says "this account is three days old" just as well as it says "this account is automated." It was my original example when explaining why a test built from the assumption it's testing proves nothing.
The gate certified it more confidently than it certified the signal I actually trust.
The reason is in how the labels were made. My automated label means followed me within a day of signing up. Empty location tracks account newness. Those aren't two proxies for the same hidden thing — they're the same proxy wearing two names. The margin is real, the p-value is honest, and both are about the confound.
I would have read .0011 as evidence the indicator was good, if someone hadn't told me this first, in advance, in writing.
Why I can't stratify my way out
My instinct was to fix it afterward: split the human label by how it was established and see whether the circular indicators wobble when the label source changes.
They didn't. Location moved zero points across that split. The indicator I trust most moved 43. With seven accounts per arm the comparison is noise, and I'd have reported whichever direction it went as a finding.
The deeper problem isn't sample size. Stratifying after the fact means conditioning on the very thing I'm trying to detect. A bigger n makes the noise quieter and the answer no more valid. The confound was installed the moment I decided what "automated" meant.
What the seed needs is an automated signal independent of account age by construction — not a window measured from signup. I tried one: regularity in timing between accounts, which doesn't depend on how old any single account is.
confirmed-human accounts, gap before their follow (minutes):
1.0 1.4 3.7 12.5 36.1 48.9 population median: 22.3
It spans the entire suspicious band. It may still be a population-level pattern, but it can't carry a per-account label, and a per-account label is the whole job.
Cross-account content duplication is the candidate left standing — nothing about being new explains two accounts posting the same text. That's the next build, and I don't know yet whether it works.
The part I'd hand back
A gate that catches indicators with no signal is the easy half. Those are cheap to find once you look: one number beside another number, and the defect is visible.
The hard half is an indicator with plenty of signal, about the wrong thing. No amount of statistics run on a confounded seed will find it, because every test you run keeps confirming it. The arithmetic is correct. The p-value is honest. It's answering a question you didn't ask.
So here's the one I'd take to your own labels. It takes about a minute:
How was this label generated, and could the thing you're testing have produced it?
If the answer is yes, your seed and your indicator are measuring the same thing, and every result from that pairing will look like confirmation.
Mine did, at p = .001, for an indicator I had already thrown in the bin.
This is part of Stolen from the Feed — what a non-developer carries home from dev.to and actually builds. Sister series to From Zero to Ship.
Top comments (0)