If you consume IP threat feeds, you have made a caching decision whether you
meant to or not. You pulled a blocklist, you stored it, and something in your
stack decides when an entry stops counting. A TTL, a nightly refresh, a cron
that truncates a table. That number is usually picked by feel.
We had the data to check the feel, so we did. This is what the decay actually
looks like across 923,000 IP-by-feed observations in our threat corpus, and one
methodological trap that ate more than half of the raw data before we could use
it.
What "days dirty" means here
For every IP on every feed we ingest, we keep a first_seen and a last_seen.
The span between them is how long that address stayed continuously listed on
that feed — its dirty duration. The corpus behind this post is 2,048,481
IP-by-feed observations, and the observation window runs from October 2025 to
late August 2026, though most feeds joined in March 2026.
Two things to keep straight, because they are different objects that happen to
share a data type:
- An incident list records that an address did something: attacked a host, scraped a form, failed a thousand logins.
- An infrastructure list records that an address is something: a VPN egress, an open proxy, a Tor exit.
Almost everything interesting below comes from those two behaving nothing alike.
Finding 1: incident data is close to disposable
Abuse and bot listings are far more ephemeral than most cache settings assume.
| Category | Observations | Median dirty | p90 | Gone within a day |
|---|---|---|---|---|
| Abuse / DDoS reports | 554,081 | 1.0 day | 90 days | 45.8% |
| Bot listings (general) | 95,547 | 1.0 day | 85 days | 34.3% |
| Bot listings (fast-expiry source) | 14,869 | 0.0 days | 18 days | 72.0% |
Across all 664,497 incident-type observations, 44.7% were gone within a
single day. The median address on the largest abuse feed we ingest is listed
for one day and never seen again.
Note the shape, though: median 1 day but p90 of 90 days. This is not a
distribution with a meaningful average. It is two populations wearing one
label — a large mass of one-shot addresses, almost certainly dynamic
residential IPs and short-lived compromised hosts, plus a small hard core of
persistent offenders that stay listed for months. Reporting a mean here would
describe neither group.
The practical consequence is uncomfortable for anyone holding this data for
weeks: after a month, most of what you are blocking is an address that
misbehaved once, got cleaned up or got reassigned to a different subscriber,
and has been innocent ever since.
Finding 2: infrastructure data is the opposite
| Category | Observations | Median dirty | p90 | Still listed in last 7 days |
|---|---|---|---|---|
| Open proxies (community lists) | 214,551 | 5.2–23.0 days | 74–130 days | 14.6–19.0% |
| Tor exit nodes | 18,525 | 13.5 days | 164.5 days | 44.5% |
| VPN ranges (provider A) | 13,938 | 119.7 days | 147.7 days | 60.8% |
| VPN ranges (provider B) | 11,526 | 7.0 days | 119.7 days | 12.9% |
Open proxies persist for weeks rather than hours. Tor exits have a median
around two weeks and a very long tail — a p90 of 164 days, which fits how Tor
actually works, since a stable relay operator keeps that relay running for a
long time.
The two VPN providers disagree violently, and that is worth not smoothing over.
One shows a median of 119.7 days; the other shows 7.0 days with 47.7% gone
within a day. Same category, two orders of magnitude apart. The likely reason
is that these are two different kinds of published artifact — one provider's
list looks like a stable allocation of owned ranges, the other's looks like a
churning pool of active egress addresses. If you treat "VPN list" as a single
category with a single freshness assumption, one of those two will be wrong for
you all the time.
The caveat that ate 55% of the data
This is the part most posts like this leave out, and it is the part that
determines whether you should trust the numbers above.
The raw corpus was 2,048,481 observations. We used 923,037. The other
1,125,444 — 54.9% — cannot answer this question at all, because they are
right-censored.
Right-censoring is the survival-analysis trap that lies in wait for anyone
measuring durations from a fixed window. If an address was already listed when
we started watching and is still listed when we stop, its measured duration is
not its lifetime. It is the length of our observation window.
The tell is unmistakable once you look for it. Seven feeds reported a median
dirty duration exactly equal to their observation span, with a p90 identical to
the median and 0% of addresses decaying:
- One large VPN provider list: 1,119,821 observations, median 147.7 days, observation window 148 days.
- Six community proxy lists: median and p90 both 151.4 days, window 151 days, 0.0% gone within a day, 100% still listed in the last 7 days.
A distribution where the median equals the p90 equals the window length, with
zero decay, is not a finding about how long proxies stay dirty. It is a feed
whose contents did not change across the entire window. For the six small proxy
lists, the most probable explanation is simply that those upstream sources have
stopped being updated — abandoned public lists are common, and they degrade
silently because a stale list still returns 200 OK and still parses.
That single VPN list is 55% of the raw corpus on its own. Had we reported the
naive aggregate, the headline would have been "the median IP stays dirty for
148 days," which is not merely wrong but almost exactly backwards from the
finding for incident data. The honest statement about those censored feeds is
that their true median is unknown and greater than the window — not that it
is 148 days.
What to do with this
Three things follow, none of which require our product.
Do not run one TTL across categories. A retention window tuned to abuse
data is roughly an order of magnitude too short for VPN and proxy
infrastructure. Tuned to infrastructure, it is far too long for abuse data, and
you will spend that surplus blocking reassigned addresses. Segment your
retention by what the list is asserting.
Expire incident data aggressively and weight it by recency. Given a median
of one day, an abuse hit from three weeks ago is weak evidence. It is not zero
evidence — the p90 tail is real — but it should not carry the same weight as a
hit from this morning. Recency belongs in the score, not just in the cache
eviction policy.
Check that your feeds are still alive. The six frozen lists are the cheap
lesson here. A feed that stops updating does not throw an error; it just
quietly becomes a fixed set of increasingly stale addresses that your system
keeps trusting. If the contents of a list have not changed in a month, that is
a monitoring alert, not a stable feed.
Method notes
Durations are last_seen - first_seen per IP per feed, computed per feed and
then grouped by category rather than pooled, since pooling lets the largest
feed dominate the aggregate — which is precisely how the censoring problem
above would have gone unnoticed. Medians and p90s are per-feed; ranges are
shown where feeds in a category disagree. All figures are aggregates over our
threat corpus; no customer traffic, no individual addresses.
We build IPASIS,
a real-time IP and email risk API, which is why we have this corpus sitting
around. If you want to poke at the scoring directly, the
OpenAPI spec is public.
Happy to share the per-category breakdown in more detail if anyone wants to
replicate this against their own feeds — the method is simple enough that you
can run it on whatever you are already ingesting, and the censoring check is
the only part worth being careful about.
Top comments (1)
The category split bites on source agreement too, not just TTL. Some cheap proxy sources blanket-flag whole AWS and Hetzner ranges, so two of them agreeing on a datacenter IP is one opinion counted twice. We ended up demanding independent specialist consensus before flagging datacenter space, while a single hit is enough on residential. Known VPN ASNs skip the vote.