One vendor was deciding our residential proxy verdicts, and I couldn't grade it against itself

My IP reputation checker has a rule that moves scores more than the weighted average ever does. When enough sources say "this address is a proxy or VPN", the score gets a hard floor of 65, which puts it in the caution band. How many sources count as "enough" depends on the address type:

const need = isDatacenter ? 2 : 1; // dedicated votes, ip-api excluded
const realProxy = dedicatedVotes >= need || !!knownAnonAsn;
if (realProxy) floors.push({ key: "proxyVpn", floor: 65 });

Datacenter ranges need two independent vendors, because a couple of feeds like to tag whole cloud blocks as proxies. Residential addresses need one, on the theory that residential proxies are rare but real when you do see them. I picked both numbers in August and never measured either.

What the logs said

Last week I pulled the per-lookup vote signatures we already keep. They record which source said what and the final verdict, with no IPs, ASNs or countries. Over a 15-day window the residential one-vote row was the uncomfortable one: 1,256 verdicts pushed into the caution band by a single vote, and in 96% of them that vote came from the same vendor.

So "one dedicated vote" really meant one vendor deciding, alone, for people on home connections. Nothing in the pipeline pushed back, because almost nothing else was voting at the time. One source sat behind a quota gate, another was dormant, and our own offline list only votes when an address is on it.

The datacenter side looked fine. The two-vote rule was absorbing that vendor's habit of flagging entire cloud ranges, which is what it was written for.

The yardstick I almost used

The obvious check is to compare the single vote against our final verdict. On a residential address with one vote, the final verdict is that vote, so the comparison returns 100% and tells you nothing.

Calibration needs a field the thing being graded can't write to. We had just wired in two more vendors, IPHub and vpnapi.io, which are unrelated to each other and to the one under test. The check became: on lookups where only that vendor said "proxy", what did the other two say?

for (const row of rows) {
  if (state(row, SUBJECT) !== ANON) continue;       // only its solo proxy calls
  const votes = YARDSTICKS.map((s) => state(row, s));
  if (votes.includes(ANON)) t.confirmed += row.n;
  else if (votes.includes(CLEAN)) t.refuted += row.n;
  else t.absent += row.n;                            // no opinion: not in the denominator
}
const rate = t.confirmed / (t.confirmed + t.refuted);

I care most about the absent bucket. If both yardsticks time out, or only report Tor or abuse history, they said nothing about proxies. Counting those rows as refuted would drag the rate down for no reason.

The decision rule is coarse on purpose. For residential, 70% confirmation or better keeps one vote, under 40% tightens it to two, and anything in between holds the current value. The datacenter rule can only get stricter, from two votes to three.

Letting a script touch a scoring rule

A weekly job rewriting a threshold that decides user-facing verdicts is a risk of its own, so three guardrails live in code. Each threshold has hard bounds (residential 1 or 2, datacenter 2 or 3). Nothing moves until confirmed plus refuted reaches 500 samples. One run can shift a threshold by one step at most.

My first version still got it wrong. The weekly build wrote the new JSON and deployed it, then the next ordinary push redeployed whatever file sat in the repo. Production would have flipped between the two, and nobody could have said which threshold was live on a given day.

Now the job only proposes. It writes the result to a temp file, and if a threshold would move, CI opens an issue with the full JSON and the evidence. Production reads the committed file and nothing else. I either paste the proposal into the repo or close the issue.

Where it stands, and what it costs

The first run changed nothing. The 500-sample gate held both thresholds, because the two yardstick vendors went live the same day the job did. The published calibration file still shows zero samples and the August values, and until data piles up, residential proxy verdicts remain one vendor's call.

The yardstick has a blind spot I can't measure yet. "Confirmed" means two other commercial feeds agreed. It doesn't mean anyone saw a real proxy, and if all three vendors buy from the same upstream list, agreement costs nothing.

What I actually got out of this week is a smaller thing than a better threshold: I know which number in the scoring code carries the most weight, and I know it had never been checked against anything.

I built ipok.io. The current calibration file is public at ipok.io/api/data/calibration if you want to watch the rate fill in.