DEV Community

Cover image for Zero asset requests was the tell: finding a scraper in 50 seconds of logs
Ava Bagherzadeh
Ava Bagherzadeh

Posted on

Zero asset requests was the tell: finding a scraper in 50 seconds of logs

#devops #webdev #cloudflare #database

Four alarms fired overnight. Two database contention pages, one read-replica collapse, and a batch of metrics that came back empty.

Four alarms, one cause, and the cause was not anything we shipped.

The wrong first instinct

The reflex when four alarms land at once is to open the deploy log and start reading diffs. I did that for a few minutes. It is almost always wasted time, and there is a cheaper question that settles it.

Did request volume change?

workersInvocationsAdaptive, grouped by script, per hour
  prod-api  3.6K/hr  ->  18K/hr   at 01:00Z
  prod-web  3.6K/hr  ->  17K/hr   at 01:00Z
Enter fullscreen mode Exit fullscreen mode

Both workers stepped up five times at the same minute and stayed there. A deploy does not do that. Traffic does. One query, and the entire code-review lane is closed.

The database side matched:

d1AnalyticsAdaptiveGroups, by databaseRole
  primary reads  15-28K/hr  ->  sustained 37-46K/hr  at 01:00Z
Enter fullscreen mode Exit fullscreen mode

D1's primary is single-threaded. Roughly ten queries a second is about 36K an hour, so we were sitting above the ceiling. Everything downstream, the refusals, the replica page, the empty metrics, is one queue backing up.

The 50 seconds that named it 

wrangler tail aiapplyd-prod-web --format json
Enter fullscreen mode Exit fullscreen mode

Fifty seconds of live requests. 187 of them. What was in there:

  • 155 of 187 from one hosting provider's VPS range, across eight or more addresses, geolocated to a single country.
  • Plain desktop Chrome user agents, rotating across five major versions plus a macOS build. No bot user agent anywhere. Nothing in robots.txt would have stopped it, because it never asked.
  • 152 requests to /jobs/{id} and 29 to /companies/{slug}. Nothing else.

187 requests: 152 job pages, 29 company pages, zero assets

That last line is the whole diagnosis.

Not one CSS file. Not one JS chunk. No font. No favicon.

A browser cannot render a page without them. A human cannot browse without a browser. So whatever this was, it was reading HTML and discarding it, at a rate of roughly four pages a second, walking two URL families in ID order.

The rotating user agent is what makes this worth writing down. It is a deliberate signal that the client wants to look human. The asset requests are the signal it forgot to fake, and they are free to check because they are already in your logs.

Why caching was not the answer

The first three suggestions in the room were all cache-shaped: longer TTL, more edge caching, a stale-while-revalidate window.

None of them help. The two families being walked hold hundreds of thousands of distinct IDs. Every request is a unique URL, on a first visit, at a rate the origin cannot absorb. A cache that never gets a second request is an extra hop. You cannot cache your way out of an enumeration.

What does work, in order of how quickly it can be deployed:

  1. Rate limit by ASN or by IP prefix, not by user agent. The user agent is attacker-controlled. The network path is not.
  2. Serve the enumerable families cheaply. If they are not indexable anyway, they do not need a database read per request.
  3. Verified-bot checks are not free. On our plan, the verified-bot field is enterprise-only. We reached for it, we could not have it, and knowing that in advance would have saved twenty minutes.

The two instrumentation lessons

Tail the worker that is actually being hit. Our API worker sees requests from our own frontend worker, which forwards without the original user agent. Tailing the API would have shown a wall of identical internal requests and nothing else. The client's fingerprint only exists at the true front door. Name the process that receives the traffic, then tail that one.

And check the alarm's window before you believe it. One of these alarms re-fired the next morning at full severity for an incident that had been fixed the previous afternoon, because it computed its rate over a rolling 24 hours. The burst was inside the window. The verdict was stale.

A rate wants a wide window. A verdict wants a recency gate. If you only have one window, you get paged for your own fixes, and the day you stop trusting the page is the day it is real.

I build AI Applyd. We scrape job listings ourselves, on a strict budget and only from sources that allow it, which is exactly why I recognise the shape of a client that does not read your CSS.

Top comments (1)

Collapse
 
szp2005 profile image
szp2005 •

Prefix-wide limits need a criterion or you'll blackhole a whole cloud. Two things worked for us: only trust a proxy/VPN verdict on datacenter IPs when two independent reputation sources agree, since a single source flags entire AWS and Hetzner ranges as proxy. Then grade the /24 by share of already-flagged neighbours. Under 10%, limit the IP, not the block.